[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:38 dist-test-slave-rh5v krb5kdc[25147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:38 dist-test-slave-rh5v krb5kdc[25147](info): set up 2 sockets
Jun 20 12:13:38 dist-test-slave-rh5v krb5kdc[25147](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 12:13:39.758694 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:13:39 dist-test-slave-rh5v krb5kdc[25147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:13:39.902393 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44427
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:44427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:40.061923 25160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:40.062371 25160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:40.062443 25160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:40.067730 25160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:13:40.067888 25160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:40.067921 25160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:40.067946 25160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:13:40.067968 25160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:13:40.075199 25160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:44427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:40.077636 25160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:40.079771 25160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:40.096004 25165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:40.096171 25160 server_base.cc:1034] running on GCE node
W20260620 12:13:40.097193 25168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:40.097573 25166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:40.098711 25160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:40.100750 25160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:40.104147 25160 hybrid_clock.cc:648] HybridClock initialized: now 1781957620104069 us; error 143 us; skew 500 ppm
Jun 20 12:13:40 dist-test-slave-rh5v krb5kdc[25147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957620, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:40.108193 25160 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:13:40.109794 25160 webserver.cc:460] Webserver started at http://127.24.141.62:38209/ using document root <none> and password file <none>
I20260620 12:13:40.110613 25160 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:40.110702 25160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:40.111104 25160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:40.114436 25160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "52d14b5af82a2e39aaac9fd760b4242d"
server_key_iv: "6908de2e4058276ebe2d1da1fe6aa24a"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.115264 25160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "52d14b5af82a2e39aaac9fd760b4242d"
server_key_iv: "6908de2e4058276ebe2d1da1fe6aa24a"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.121778 25160 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 12:13:40.126513 25175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:40.129159 25160 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:13:40.129441 25160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "52d14b5af82a2e39aaac9fd760b4242d"
server_key_iv: "6908de2e4058276ebe2d1da1fe6aa24a"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.129631 25160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:40.144629 25160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:40.149408 25160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:40.149684 25160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:40.160038 25160 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:44427
I20260620 12:13:40.160068 25227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:44427 every 8 connection(s)
I20260620 12:13:40.161613 25160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:13:40.163691 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 25160
I20260620 12:13:40.163919 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:13:40.166356 25140 external_mini_cluster.cc:1428] Setting key 78fb6170d20004138086b5fd4a9e0e07
I20260620 12:13:40.169113 25228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:40.180610 25228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a: Bootstrap starting.
I20260620 12:13:40.186141 25228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:40.188195 25228 log.cc:824] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:40.192243 25228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a: No bootstrap required, opened a new log
Jun 20 12:13:40 dist-test-slave-rh5v krb5kdc[25147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:40.198372 25228 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44427 } }
I20260620 12:13:40.198907 25228 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:40.199043 25228 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e36d5cb7934bec9eb6b5e5b361c68a, State: Initialized, Role: FOLLOWER
I20260620 12:13:40.200088 25228 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44427 } }
I20260620 12:13:40.200363 25228 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:13:40.200522 25228 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:13:40.200716 25228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:40.202203 25228 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44427 } }
I20260620 12:13:40.202804 25228 leader_election.cc:302] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e36d5cb7934bec9eb6b5e5b361c68a; no voters:
I20260620 12:13:40.203274 25228 leader_election.cc:288] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:13:40.203411 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:40.174177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.24.141.62:44427)
0620 12:13:40.174708 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:40.174734 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:40.174790 (+ 56us) server_negotiation.cc:407] Connection header received
0620 12:13:40.179970 (+ 5180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:40.179996 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:40.180505 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:40.180986 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:40.185013 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.186610 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.188249 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.188860 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.189049 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:40.195205 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:40.195214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:40.195236 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:40.195285 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:40.197363 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:40.198409 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:40.198418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:40.198440 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:40.198697 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:40.199444 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:40.199451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:40.199453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:40.200355 (+ 902us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:40.200973 (+ 618us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:40.201606 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 12:13:40.202061 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":143,"threads_started":1}
I20260620 12:13:40.203512 25233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:40.203913 25233 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [term 1 LEADER]: Becoming Leader. State: Replica: f0e36d5cb7934bec9eb6b5e5b361c68a, State: Running, Role: LEADER
I20260620 12:13:40.204487 25233 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44427 } }
I20260620 12:13:40.205186 25228 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:13:40.206501 25233 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44427 } } }
I20260620 12:13:40.206663 25233 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:40.206948 25233 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e36d5cb7934bec9eb6b5e5b361c68a. Latest consensus state: current_term: 1 leader_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e36d5cb7934bec9eb6b5e5b361c68a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44427 } } }
I20260620 12:13:40.207104 25233 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a [sys.catalog]: This master's current role is: LEADER
W20260620 12:13:40.212756 25248 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:13:40.212882 25248 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:13:40.212996 25242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:13:40.214244 25242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:13:40.223800 25242 catalog_manager.cc:1347] Generated new cluster ID: 19de171803d54051b1e35703402433ed
I20260620 12:13:40.223910 25242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:13:40.232429 25242 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:13:40.233793 25242 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:13:40.241918 25242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e36d5cb7934bec9eb6b5e5b361c68a: Generated new TSK 0
I20260620 12:13:40.242904 25242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:13:40.285707 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44427
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:40.447486 25254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:40.447829 25254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:40.447877 25254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:40.452188 25254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:40.452286 25254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:40.452360 25254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:13:40.458424 25254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:40.459923 25254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:40.461237 25254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:40.474399 25260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:40.474371 25259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:40.474741 25262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:40.474901 25254 server_base.cc:1034] running on GCE node
I20260620 12:13:40.475479 25254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:40.476392 25254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:40.477608 25254 hybrid_clock.cc:648] HybridClock initialized: now 1781957620477572 us; error 44 us; skew 500 ppm
Jun 20 12:13:40 dist-test-slave-rh5v krb5kdc[25147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957620, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:40.481470 25254 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:13:40.482669 25254 webserver.cc:460] Webserver started at http://127.24.141.1:35405/ using document root <none> and password file <none>
I20260620 12:13:40.483429 25254 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:40.483512 25254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:40.483757 25254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:40.486357 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "a1b520f3d2d74dcfa7ae8d8d461ae614"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "a262367f524093b6aa2a0765f1ec2293"
server_key_iv: "b454f5b15d45ea3ac2ba6e01838f4ba3"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.487015 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "a1b520f3d2d74dcfa7ae8d8d461ae614"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "a262367f524093b6aa2a0765f1ec2293"
server_key_iv: "b454f5b15d45ea3ac2ba6e01838f4ba3"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.492306 25254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:13:40.495820 25269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:40.497860 25254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:13:40.498051 25254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "a1b520f3d2d74dcfa7ae8d8d461ae614"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "a262367f524093b6aa2a0765f1ec2293"
server_key_iv: "b454f5b15d45ea3ac2ba6e01838f4ba3"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.498202 25254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:40.511862 25254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:40.515609 25254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:40.515831 25254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:40.516616 25254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:40.517958 25254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:40.518020 25254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:40.518080 25254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:40.518106 25254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:40.530023 25254 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:35403
I20260620 12:13:40.530046 25382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:35403 every 8 connection(s)
I20260620 12:13:40.531162 25254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:13:40.535699 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 25254
I20260620 12:13:40.535826 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:13:40.536098 25140 external_mini_cluster.cc:1428] Setting key 88481c55786ab99c80002d4fdbc608b9
Jun 20 12:13:40 dist-test-slave-rh5v krb5kdc[25147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957620, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:13:40.550309 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:40.534143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:33427 (local address 127.24.141.62:44427)
0620 12:13:40.534341 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:40.534347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:40.535666 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 12:13:40.537585 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:40.537589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:40.537655 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:40.537739 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:40.539375 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.539964 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.541574 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.541941 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.542108 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:40.545697 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:40.545701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:40.545704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:40.545739 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:40.547226 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:40.548039 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:40.548043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:40.548044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:40.548133 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:40.548683 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:40.548687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:40.548688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:40.548934 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:40.549036 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:40.549935 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 12:13:40.550098 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:13:40.551407 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:40.534751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44427 (local address 127.24.141.1:33427)
0620 12:13:40.535447 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:40.535515 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:40.537314 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:40.537981 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:40.537991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:40.538491 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:40.539158 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:40.539175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.540122 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.540126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:40.541409 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:40.541422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.542189 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.542193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:40.542418 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:40.543153 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:40.543172 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:40.545367 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:40.547404 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:40.547413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:40.547432 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:40.547868 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:40.548262 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:40.548265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:40.548267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:40.548502 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:40.549117 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:40.549126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:40.549582 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 12:13:40.549806 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 12:13:40.550208 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":171,"threads_started":1}
I20260620 12:13:40.553225 25383 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44427
I20260620 12:13:40.553591 25383 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:40.554309 25383 heartbeater.cc:511] Master 127.24.141.62:44427 requested a full tablet report, sending...
I20260620 12:13:40.556761 25188 ts_manager.cc:194] Registered new tserver with Master: a1b520f3d2d74dcfa7ae8d8d461ae614 (127.24.141.1:35403)
I20260620 12:13:40.558419 25188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:33427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:13:40.566524 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44427
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:40.701479 25388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:40.701808 25388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:40.701874 25388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:40.706334 25388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:40.706434 25388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:40.706521 25388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:13:40.713183 25388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:40.714558 25388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:40.715765 25388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:40.728866 25396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:40.728866 25393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:40.728866 25394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:40.729713 25388 server_base.cc:1034] running on GCE node
I20260620 12:13:40.730319 25388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:40.731249 25388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:40.734624 25388 hybrid_clock.cc:648] HybridClock initialized: now 1781957620734567 us; error 75 us; skew 500 ppm
Jun 20 12:13:40 dist-test-slave-rh5v krb5kdc[25147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957620, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:40.739322 25388 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:13:40.740733 25388 webserver.cc:460] Webserver started at http://127.24.141.2:39279/ using document root <none> and password file <none>
I20260620 12:13:40.741676 25388 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:40.741833 25388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:40.742190 25388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:40.745672 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "d002a0b4c48a407db73d04cb909e730f"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "24677766be87967e59d77744fb422641"
server_key_iv: "99c9afdd091b10631095bf75c1e2137a"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.746609 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "d002a0b4c48a407db73d04cb909e730f"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "24677766be87967e59d77744fb422641"
server_key_iv: "99c9afdd091b10631095bf75c1e2137a"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.752882 25388 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 12:13:40.757100 25403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:40.759466 25388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:13:40.759698 25388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "d002a0b4c48a407db73d04cb909e730f"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "24677766be87967e59d77744fb422641"
server_key_iv: "99c9afdd091b10631095bf75c1e2137a"
server_key_version: "encryptionkey@0"
I20260620 12:13:40.759862 25388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:40.774718 25388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:40.779807 25388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:40.780016 25388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:40.780684 25388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:40.781762 25388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:40.781821 25388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:40.781883 25388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:40.781909 25388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:40.792122 25388 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:34727
I20260620 12:13:40.792171 25516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:34727 every 8 connection(s)
I20260620 12:13:40.793256 25388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:13:40.795182 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 25388
I20260620 12:13:40.795311 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:13:40.795634 25140 external_mini_cluster.cc:1428] Setting key 0e4d5d4c94adbc5473fd5d6ed1680c6b
Jun 20 12:13:40 dist-test-slave-rh5v krb5kdc[25147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957620, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:13:40.811937 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:40.795902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:59247 (local address 127.24.141.62:44427)
0620 12:13:40.796098 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:40.796103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:40.797371 (+ 1268us) server_negotiation.cc:407] Connection header received
0620 12:13:40.799349 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:40.799353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:40.799425 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:40.799521 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:40.801429 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.802075 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.803651 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.803884 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.803981 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:40.807300 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:40.807304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:40.807307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:40.807337 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:40.808922 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:40.809696 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:40.809699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:40.809701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:40.809812 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:40.810411 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:40.810415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:40.810416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:40.810637 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:40.810726 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:40.811629 (+ 903us) server_negotiation.cc:299] Negotiation successful
0620 12:13:40.811790 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 12:13:40.813302 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:40.796394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44427 (local address 127.24.141.2:59247)
0620 12:13:40.797142 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:40.797229 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:40.798995 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:40.799820 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:40.799833 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:40.800464 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:40.801212 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:40.801233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.802293 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.802299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:40.803464 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:40.803485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:40.804056 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:40.804060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:40.804270 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:40.804954 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:40.804988 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:40.807056 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:40.809106 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:40.809114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:40.809140 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:40.809532 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:40.809962 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:40.809967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:40.809969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:40.810252 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:40.810757 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:40.810767 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:40.811269 (+ 502us) client_negotiation.cc:769] Sending connection context
0620 12:13:40.811530 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 12:13:40.812066 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":159,"threads_started":1}
I20260620 12:13:40.814999 25517 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44427
I20260620 12:13:40.815510 25517 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:40.816551 25517 heartbeater.cc:511] Master 127.24.141.62:44427 requested a full tablet report, sending...
I20260620 12:13:40.818358 25188 ts_manager.cc:194] Registered new tserver with Master: d002a0b4c48a407db73d04cb909e730f (127.24.141.2:34727)
I20260620 12:13:40.819124 25188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:59247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:13:40.828029 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44427
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:40.963267 25522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:40.963609 25522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:40.963658 25522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:40.968094 25522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:40.968185 25522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:40.968254 25522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:13:40.974171 25522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:40.975554 25522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:40.976888 25522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:40.989132 25530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:40.989146 25528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:40.989328 25522 server_base.cc:1034] running on GCE node
W20260620 12:13:40.989491 25527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:40.990188 25522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:40.991063 25522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:40.992322 25522 hybrid_clock.cc:648] HybridClock initialized: now 1781957620992287 us; error 45 us; skew 500 ppm
Jun 20 12:13:40 dist-test-slave-rh5v krb5kdc[25147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957620, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:40.996197 25522 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:13:40.997516 25522 webserver.cc:460] Webserver started at http://127.24.141.3:41327/ using document root <none> and password file <none>
I20260620 12:13:40.998246 25522 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:40.998328 25522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:40.998574 25522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:41.001417 25522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "c135ce78a0074d73b21cffe46f9eb5d8"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "b5149f8301252fb9ab24c417fcd6eeef"
server_key_iv: "1f25987a58ebce4edfdcd34a530cf030"
server_key_version: "encryptionkey@0"
I20260620 12:13:41.002247 25522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "c135ce78a0074d73b21cffe46f9eb5d8"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "b5149f8301252fb9ab24c417fcd6eeef"
server_key_iv: "1f25987a58ebce4edfdcd34a530cf030"
server_key_version: "encryptionkey@0"
I20260620 12:13:41.007925 25522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:13:41.011260 25537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:41.013048 25522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:41.013331 25522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "c135ce78a0074d73b21cffe46f9eb5d8"
format_stamp: "Formatted at 2026-06-20 12:13:40 on dist-test-slave-rh5v"
server_key: "b5149f8301252fb9ab24c417fcd6eeef"
server_key_iv: "1f25987a58ebce4edfdcd34a530cf030"
server_key_version: "encryptionkey@0"
I20260620 12:13:41.013471 25522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:41.035954 25522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:41.040756 25522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:41.041024 25522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:41.041796 25522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:41.042856 25522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:41.042912 25522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:41.042969 25522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:41.042999 25522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:41.054884 25522 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:38775
I20260620 12:13:41.054960 25650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:38775 every 8 connection(s)
I20260620 12:13:41.055958 25522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:13:41.057330 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 25522
I20260620 12:13:41.057459 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:13:41.057729 25140 external_mini_cluster.cc:1428] Setting key 9f3eb5a92b0f0593810eee3dd6fcc4c5
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957620, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:41.073601 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:41.058486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:39151 (local address 127.24.141.62:44427)
0620 12:13:41.058684 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:41.058689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:41.060219 (+ 1530us) server_negotiation.cc:407] Connection header received
0620 12:13:41.061277 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:41.061279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:41.061339 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:41.061418 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:41.062724 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.063308 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.064557 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.064791 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.064879 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:41.068040 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:41.068045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:41.068047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:41.068087 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:41.070009 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:41.070852 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:41.070855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:41.070857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:41.070959 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:41.071776 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:41.071780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:41.071782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:41.072203 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:41.072319 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:41.073217 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 12:13:41.073402 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:13:41.074828 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:41.059095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44427 (local address 127.24.141.3:39151)
0620 12:13:41.059975 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:41.060062 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:41.061003 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:41.061593 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:41.061601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:41.061997 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:41.062573 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:41.062588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.063435 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.063438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:41.064431 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:41.064443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.064933 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.064936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:41.065175 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:41.065858 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:41.065878 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:41.067757 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:41.070230 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:41.070239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:41.070264 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:41.070718 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:41.071225 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:41.071231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:41.071234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:41.071612 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:41.072387 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:41.072397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:41.072858 (+ 461us) client_negotiation.cc:769] Sending connection context
0620 12:13:41.073172 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 12:13:41.073601 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":179,"threads_started":1}
I20260620 12:13:41.076423 25651 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44427
I20260620 12:13:41.076728 25651 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:41.077425 25651 heartbeater.cc:511] Master 127.24.141.62:44427 requested a full tablet report, sending...
I20260620 12:13:41.078856 25188 ts_manager.cc:194] Registered new tserver with Master: c135ce78a0074d73b21cffe46f9eb5d8 (127.24.141.3:38775)
I20260620 12:13:41.079531 25188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:39151
I20260620 12:13:41.087266 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:13:41.117949 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:41.106640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.24.141.1:35403)
0620 12:13:41.106938 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:41.106947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:41.107228 (+ 281us) server_negotiation.cc:407] Connection header received
0620 12:13:41.107348 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:41.107357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:41.107600 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:41.107765 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:41.108610 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.109504 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.110496 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.110938 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.111049 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:41.113650 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:41.113667 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:41.113687 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:41.113727 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:41.115679 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:41.116226 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:41.116236 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:41.116238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:41.116378 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:41.116728 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:41.116733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:41.116735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:41.117167 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:41.117390 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:41.117610 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 12:13:41.117777 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
W20260620 12:13:41.119335 25297 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60520
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:13:41.150928 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:41.141317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.24.141.1:35403)
0620 12:13:41.141528 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:41.141533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:41.141678 (+ 145us) server_negotiation.cc:407] Connection header received
0620 12:13:41.141834 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:41.141837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:41.141909 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:41.141995 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:41.142906 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.143468 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.144480 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:41.144701 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:41.144775 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:41.147507 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:41.147513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:41.147515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:41.147557 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:41.149027 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:41.149496 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:41.149499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:41.149500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:41.149585 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:41.149846 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:41.149850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:41.149851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:41.150198 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:41.150348 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:41.150632 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 12:13:41.150790 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:13:41.152308 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 25254
I20260620 12:13:41.158522 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 25388
I20260620 12:13:41.165978 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 25522
I20260620 12:13:41.175189 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 25160
2026-06-20T12:13:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2416 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25665](info): set up 2 sockets
Jun 20 12:13:41 dist-test-slave-rh5v krb5kdc[25665](info): commencing operation
krb5kdc: starting...
W20260620 12:13:42.133615 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:13:42 dist-test-slave-rh5v krb5kdc[25665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:13:42.272902 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38273
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:38273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:42.404395 25678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:42.404727 25678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:42.404773 25678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:42.409251 25678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:13:42.409350 25678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:42.409370 25678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:42.409386 25678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:13:42.409401 25678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:13:42.415722 25678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:38273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:42.417032 25678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:42.418344 25678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:42.428624 25683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:42.428615 25686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:42.428624 25684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:42.429337 25678 server_base.cc:1034] running on GCE node
I20260620 12:13:42.430080 25678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:42.431526 25678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:42.432765 25678 hybrid_clock.cc:648] HybridClock initialized: now 1781957622432738 us; error 55 us; skew 500 ppm
Jun 20 12:13:42 dist-test-slave-rh5v krb5kdc[25665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957622, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:42.436508 25678 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:13:42.438035 25678 webserver.cc:460] Webserver started at http://127.24.141.62:33601/ using document root <none> and password file <none>
I20260620 12:13:42.438956 25678 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:42.439066 25678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:42.439324 25678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:42.441944 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "9eda659d14a7448b8c68d393c9a5f041"
format_stamp: "Formatted at 2026-06-20 12:13:42 on dist-test-slave-rh5v"
server_key: "0b435ead3c156134767fd1e0db958f97"
server_key_iv: "8b811ede123f74d23986a500ca942497"
server_key_version: "encryptionkey@0"
I20260620 12:13:42.442597 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "9eda659d14a7448b8c68d393c9a5f041"
format_stamp: "Formatted at 2026-06-20 12:13:42 on dist-test-slave-rh5v"
server_key: "0b435ead3c156134767fd1e0db958f97"
server_key_iv: "8b811ede123f74d23986a500ca942497"
server_key_version: "encryptionkey@0"
I20260620 12:13:42.447695 25678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:13:42.451079 25693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:42.453425 25678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:13:42.453632 25678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "9eda659d14a7448b8c68d393c9a5f041"
format_stamp: "Formatted at 2026-06-20 12:13:42 on dist-test-slave-rh5v"
server_key: "0b435ead3c156134767fd1e0db958f97"
server_key_iv: "8b811ede123f74d23986a500ca942497"
server_key_version: "encryptionkey@0"
I20260620 12:13:42.453795 25678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:42.479889 25678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:42.483474 25678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:42.483686 25678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:42.491657 25678 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:38273
I20260620 12:13:42.491708 25745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:38273 every 8 connection(s)
I20260620 12:13:42.492761 25678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:13:42.496696 25746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:42.501881 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 25678
I20260620 12:13:42.502044 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:13:42.502344 25140 external_mini_cluster.cc:1428] Setting key 21697487163f4b1e5c55fbcaf1bfa5bd
I20260620 12:13:42.504935 25746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041: Bootstrap starting.
I20260620 12:13:42.508404 25746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:42.509477 25746 log.cc:824] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:42.512413 25746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041: No bootstrap required, opened a new log
Jun 20 12:13:42 dist-test-slave-rh5v krb5kdc[25665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:42.518028 25746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eda659d14a7448b8c68d393c9a5f041" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38273 } }
I20260620 12:13:42.518409 25746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:42.518501 25746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eda659d14a7448b8c68d393c9a5f041, State: Initialized, Role: FOLLOWER
I20260620 12:13:42.519164 25746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eda659d14a7448b8c68d393c9a5f041" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38273 } }
I20260620 12:13:42.519373 25746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:13:42.519469 25746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:13:42.519578 25746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:42.520895 25746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eda659d14a7448b8c68d393c9a5f041" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38273 } }
I20260620 12:13:42.521435 25746 leader_election.cc:302] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eda659d14a7448b8c68d393c9a5f041; no voters:
I20260620 12:13:42.521731 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:42.504280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.24.141.62:38273)
0620 12:13:42.504828 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:42.504847 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:42.504897 (+ 50us) server_negotiation.cc:407] Connection header received
0620 12:13:42.505839 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:42.505862 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:42.506332 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:42.506702 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:42.507832 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:42.509486 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:42.510689 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:42.511312 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:42.511478 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:42.514856 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:42.514866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:42.514891 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:42.514948 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:42.517255 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:42.517941 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:42.517948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:42.517955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:42.518128 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:42.518607 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:42.518611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:42.518613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:42.519284 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:42.519533 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:42.520095 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 12:13:42.520512 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":151,"threads_started":1}
I20260620 12:13:42.521858 25746 leader_election.cc:288] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:13:42.522202 25751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:42.522461 25751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [term 1 LEADER]: Becoming Leader. State: Replica: 9eda659d14a7448b8c68d393c9a5f041, State: Running, Role: LEADER
I20260620 12:13:42.522954 25751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eda659d14a7448b8c68d393c9a5f041" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38273 } }
I20260620 12:13:42.523753 25746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:13:42.525722 25753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eda659d14a7448b8c68d393c9a5f041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eda659d14a7448b8c68d393c9a5f041" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38273 } } }
I20260620 12:13:42.525933 25753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:42.525837 25752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eda659d14a7448b8c68d393c9a5f041. Latest consensus state: current_term: 1 leader_uuid: "9eda659d14a7448b8c68d393c9a5f041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eda659d14a7448b8c68d393c9a5f041" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38273 } } }
I20260620 12:13:42.526162 25752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041 [sys.catalog]: This master's current role is: LEADER
W20260620 12:13:42.530184 25766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:13:42.530311 25766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:13:42.530421 25765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:13:42.531337 25765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:13:42.538470 25765 catalog_manager.cc:1347] Generated new cluster ID: e0be9ab12a9b48a3a1b587b2ce229ca6
I20260620 12:13:42.538583 25765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:13:42.550982 25765 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:13:42.552131 25765 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:13:42.560986 25765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eda659d14a7448b8c68d393c9a5f041: Generated new TSK 0
I20260620 12:13:42.561882 25765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:13:42.602701 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38273
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:42.733131 25772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:42.733469 25772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:42.733532 25772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:42.737794 25772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:42.737879 25772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:42.737964 25772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:13:42.745944 25772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:42.747766 25772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:42.749097 25772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:42.761905 25777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:42.761979 25772 server_base.cc:1034] running on GCE node
W20260620 12:13:42.761925 25778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:42.761940 25780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:42.762704 25772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:42.763509 25772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:42.764719 25772 hybrid_clock.cc:648] HybridClock initialized: now 1781957622764685 us; error 49 us; skew 500 ppm
Jun 20 12:13:42 dist-test-slave-rh5v krb5kdc[25665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957622, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:42.767827 25772 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:13:42.768960 25772 webserver.cc:460] Webserver started at http://127.24.141.1:42399/ using document root <none> and password file <none>
I20260620 12:13:42.769652 25772 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:42.769726 25772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:42.770069 25772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:42.772778 25772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "a33829aea76348cb883fc1e2e54b3ff2"
format_stamp: "Formatted at 2026-06-20 12:13:42 on dist-test-slave-rh5v"
server_key: "674cff77d8318fbe427a88ca9173dc3d"
server_key_iv: "3efa1269c4a3a1b7f763e375607f7c8e"
server_key_version: "encryptionkey@0"
I20260620 12:13:42.773535 25772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "a33829aea76348cb883fc1e2e54b3ff2"
format_stamp: "Formatted at 2026-06-20 12:13:42 on dist-test-slave-rh5v"
server_key: "674cff77d8318fbe427a88ca9173dc3d"
server_key_iv: "3efa1269c4a3a1b7f763e375607f7c8e"
server_key_version: "encryptionkey@0"
I20260620 12:13:42.778164 25772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:13:42.781359 25787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:42.783104 25772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:13:42.783285 25772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "a33829aea76348cb883fc1e2e54b3ff2"
format_stamp: "Formatted at 2026-06-20 12:13:42 on dist-test-slave-rh5v"
server_key: "674cff77d8318fbe427a88ca9173dc3d"
server_key_iv: "3efa1269c4a3a1b7f763e375607f7c8e"
server_key_version: "encryptionkey@0"
I20260620 12:13:42.783414 25772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:42.831521 25772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:42.835284 25772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:42.835496 25772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:42.836153 25772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:42.837181 25772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:42.837235 25772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:42.837296 25772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:42.837325 25772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:42.848143 25772 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:38349
I20260620 12:13:42.848248 25900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:38349 every 8 connection(s)
I20260620 12:13:42.849368 25772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:13:42.852375 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 25772
I20260620 12:13:42.852491 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:13:42.852769 25140 external_mini_cluster.cc:1428] Setting key 4d66d55df21ba5946850a2e0bb59f617
Jun 20 12:13:42 dist-test-slave-rh5v krb5kdc[25665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957622, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:13:42.868682 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:42.851724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:35959 (local address 127.24.141.62:38273)
0620 12:13:42.851952 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:42.851956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:42.853126 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 12:13:42.854830 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:42.854833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:42.854886 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:42.854981 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:42.856604 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:42.857273 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:42.859127 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:42.859527 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:42.859667 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:42.863558 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:42.863563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:42.863565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:42.863616 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:42.865333 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:42.866090 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:42.866094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:42.866096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:42.866229 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:42.866736 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:42.866740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:42.866741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:42.867001 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:42.867105 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:42.868111 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0620 12:13:42.868458 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 12:13:42.869642 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:42.852267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38273 (local address 127.24.141.1:35959)
0620 12:13:42.852898 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:42.852963 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:42.854563 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:42.855212 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:42.855222 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:42.855742 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:42.856400 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:42.856418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:42.857427 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:42.857431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:42.858945 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:42.858963 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:42.859763 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:42.859768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:42.860046 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:42.860922 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:42.860951 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:42.863187 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:42.865528 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:42.865537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:42.865558 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:42.865929 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:42.866388 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:42.866392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:42.866393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:42.866585 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:42.867169 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:42.867177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:42.867661 (+ 484us) client_negotiation.cc:769] Sending connection context
0620 12:13:42.867978 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 12:13:42.868395 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":176,"threads_started":1}
I20260620 12:13:42.871294 25901 heartbeater.cc:347] Connected to a master server at 127.24.141.62:38273
I20260620 12:13:42.871762 25901 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:42.872479 25901 heartbeater.cc:511] Master 127.24.141.62:38273 requested a full tablet report, sending...
I20260620 12:13:42.874660 25710 ts_manager.cc:194] Registered new tserver with Master: a33829aea76348cb883fc1e2e54b3ff2 (127.24.141.1:38349)
I20260620 12:13:42.875993 25710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:35959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:13:42.887990 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38273
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:43.021862 25906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:43.022264 25906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:43.022318 25906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:43.027238 25906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:43.027341 25906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:43.027424 25906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:13:43.033780 25906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:43.035270 25906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:43.036551 25906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:43.048327 25914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:43.048323 25912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:43.048314 25911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:43.048596 25906 server_base.cc:1034] running on GCE node
I20260620 12:13:43.049208 25906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:43.050055 25906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:43.051277 25906 hybrid_clock.cc:648] HybridClock initialized: now 1781957623051216 us; error 71 us; skew 500 ppm
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:43.054566 25906 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:13:43.055711 25906 webserver.cc:460] Webserver started at http://127.24.141.2:43083/ using document root <none> and password file <none>
I20260620 12:13:43.056389 25906 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:43.056465 25906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:43.056691 25906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:43.059362 25906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "24f51097f2694b3e89df61470be938eb"
format_stamp: "Formatted at 2026-06-20 12:13:43 on dist-test-slave-rh5v"
server_key: "c15f3d2e0963f5ae8ba734290409054b"
server_key_iv: "d416aba5b0fd52b07e06fc3259be763d"
server_key_version: "encryptionkey@0"
I20260620 12:13:43.060050 25906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "24f51097f2694b3e89df61470be938eb"
format_stamp: "Formatted at 2026-06-20 12:13:43 on dist-test-slave-rh5v"
server_key: "c15f3d2e0963f5ae8ba734290409054b"
server_key_iv: "d416aba5b0fd52b07e06fc3259be763d"
server_key_version: "encryptionkey@0"
I20260620 12:13:43.064898 25906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 12:13:43.068243 25921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:43.070163 25906 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:13:43.070389 25906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "24f51097f2694b3e89df61470be938eb"
format_stamp: "Formatted at 2026-06-20 12:13:43 on dist-test-slave-rh5v"
server_key: "c15f3d2e0963f5ae8ba734290409054b"
server_key_iv: "d416aba5b0fd52b07e06fc3259be763d"
server_key_version: "encryptionkey@0"
I20260620 12:13:43.070530 25906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:43.088057 25906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:43.091826 25906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:43.092023 25906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:43.092732 25906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:43.093864 25906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:43.093919 25906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:43.093979 25906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:43.094015 25906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:43.105022 25906 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:43105
I20260620 12:13:43.105105 26034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:43105 every 8 connection(s)
I20260620 12:13:43.106197 25906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:13:43.106657 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 25906
I20260620 12:13:43.106797 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:13:43.107080 25140 external_mini_cluster.cc:1428] Setting key eb7517042349df84a18d1e032e232f61
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:43.127374 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.108601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:52691 (local address 127.24.141.62:38273)
0620 12:13:43.108766 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.108770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.109949 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 12:13:43.111938 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.111942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.112009 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.112108 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:43.114006 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.114637 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.116082 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.116467 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.116567 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.122600 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:43.122605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:43.122609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:43.122654 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:43.124386 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.125373 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.125378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.125380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.125492 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.125885 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.125888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.125890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.126198 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:43.126313 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.127057 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.127210 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:13:43.128904 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.109167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38273 (local address 127.24.141.2:52691)
0620 12:13:43.109740 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:43.109800 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:43.111615 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:43.112366 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:43.112377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:43.112926 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:43.113772 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.113796 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.114816 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.114820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.115899 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.115911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.116835 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.116840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.117051 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.117741 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:43.117762 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:43.122233 (+ 4471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:43.124657 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:43.124665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:43.124686 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:43.125231 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:43.125593 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:43.125596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:43.125597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:43.125777 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:43.126348 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:43.126358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:43.126717 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 12:13:43.126979 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 12:13:43.127463 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260620 12:13:43.130504 26035 heartbeater.cc:347] Connected to a master server at 127.24.141.62:38273
I20260620 12:13:43.130816 26035 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:43.131484 26035 heartbeater.cc:511] Master 127.24.141.62:38273 requested a full tablet report, sending...
I20260620 12:13:43.132757 25710 ts_manager.cc:194] Registered new tserver with Master: 24f51097f2694b3e89df61470be938eb (127.24.141.2:43105)
I20260620 12:13:43.133486 25710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:52691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:13:43.139946 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38273
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:43.272696 26040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:43.273106 26040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:43.273180 26040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:43.277998 26040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:43.278101 26040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:43.278194 26040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:13:43.284940 26040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:43.286554 26040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:43.287926 26040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:43.300212 26045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:43.300215 26046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:43.300212 26048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:43.300624 26040 server_base.cc:1034] running on GCE node
I20260620 12:13:43.301224 26040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:43.302064 26040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:43.303287 26040 hybrid_clock.cc:648] HybridClock initialized: now 1781957623303245 us; error 60 us; skew 500 ppm
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:43.306656 26040 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:13:43.307943 26040 webserver.cc:460] Webserver started at http://127.24.141.3:33729/ using document root <none> and password file <none>
I20260620 12:13:43.308662 26040 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:43.308734 26040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:43.308983 26040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:43.311615 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "ad084ad0469448f7bc8019cdeaffcd11"
format_stamp: "Formatted at 2026-06-20 12:13:43 on dist-test-slave-rh5v"
server_key: "a9a9d3775499072730733dc278888157"
server_key_iv: "70c30ff28d64dcf3c20a2dd86cb128a9"
server_key_version: "encryptionkey@0"
I20260620 12:13:43.312263 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "ad084ad0469448f7bc8019cdeaffcd11"
format_stamp: "Formatted at 2026-06-20 12:13:43 on dist-test-slave-rh5v"
server_key: "a9a9d3775499072730733dc278888157"
server_key_iv: "70c30ff28d64dcf3c20a2dd86cb128a9"
server_key_version: "encryptionkey@0"
I20260620 12:13:43.317252 26040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 12:13:43.320583 26055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:43.322366 26040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:43.322531 26040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "ad084ad0469448f7bc8019cdeaffcd11"
format_stamp: "Formatted at 2026-06-20 12:13:43 on dist-test-slave-rh5v"
server_key: "a9a9d3775499072730733dc278888157"
server_key_iv: "70c30ff28d64dcf3c20a2dd86cb128a9"
server_key_version: "encryptionkey@0"
I20260620 12:13:43.322748 26040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:43.358417 26040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:43.362321 26040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:43.362521 26040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:43.363251 26040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:43.364255 26040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:43.364307 26040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:43.364367 26040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:43.364394 26040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:43.375641 26040 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:39993
I20260620 12:13:43.375722 26168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:39993 every 8 connection(s)
I20260620 12:13:43.376811 26040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:13:43.378813 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 26040
I20260620 12:13:43.378933 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:13:43.379245 25140 external_mini_cluster.cc:1428] Setting key 8383f95d7eb32d0d1a5917e852a2ab7d
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:43.394567 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.379352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:34747 (local address 127.24.141.62:38273)
0620 12:13:43.379542 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.379547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.381100 (+ 1553us) server_negotiation.cc:407] Connection header received
0620 12:13:43.382726 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.382730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.382795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.382875 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:43.384331 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.384873 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.386200 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.386432 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.386525 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.389728 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:43.389732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:43.389736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:43.389779 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:43.391609 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.392415 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.392419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.392421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.392511 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.392896 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.392899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.392901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.393178 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:43.393271 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.394075 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.394350 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:13:43.395671 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.379917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38273 (local address 127.24.141.3:34747)
0620 12:13:43.380886 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:43.380956 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:43.382459 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:43.383111 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:43.383121 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:43.383561 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:43.384164 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.384180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.385005 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.385008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.386055 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.386066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.386599 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.386604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.386842 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.387471 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:43.387491 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:43.389440 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:43.391809 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:43.391819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:43.391841 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:43.392258 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:43.392608 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:43.392611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:43.392613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:43.392786 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:43.393280 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:43.393289 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:43.393691 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 12:13:43.393904 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 12:13:43.394517 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":174,"threads_started":1}
I20260620 12:13:43.397142 26169 heartbeater.cc:347] Connected to a master server at 127.24.141.62:38273
I20260620 12:13:43.397504 26169 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:43.398190 26169 heartbeater.cc:511] Master 127.24.141.62:38273 requested a full tablet report, sending...
I20260620 12:13:43.399682 25710 ts_manager.cc:194] Registered new tserver with Master: ad084ad0469448f7bc8019cdeaffcd11 (127.24.141.3:39993)
I20260620 12:13:43.400421 25710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:34747
I20260620 12:13:43.406602 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:13:43.419770 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.411792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.24.141.62:38273)
0620 12:13:43.411990 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.411995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.412207 (+ 212us) server_negotiation.cc:407] Connection header received
0620 12:13:43.412368 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.412372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.412430 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.412503 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:43.413327 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.413933 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.414757 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.414975 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.415062 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.416630 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:43.416634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:43.416637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:43.416663 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:43.418032 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.418467 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.418471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.418472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.418547 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.418804 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.418808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.418809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.419065 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:43.419196 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.419445 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.419585 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 12:13:43.427223 25710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:13:43.431113 25710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:43.450759 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.443881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:38349 (local address 127.0.0.1:40966)
0620 12:13:43.444932 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:43.444948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:43.445107 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:43.446021 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:43.446024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:43.446046 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:43.446173 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.446181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.447528 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.447531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.449018 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.449028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.450097 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.450101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.450235 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.450279 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 12:13:43.450408 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 12:13:43.450577 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":274,"thread_start_us":125,"threads_started":1}
I20260620 12:13:43.450773 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.444147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40966 (local address 127.24.141.1:38349)
0620 12:13:43.444886 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.444894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.445237 (+ 343us) server_negotiation.cc:407] Connection header received
0620 12:13:43.445349 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.445356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.445649 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.445842 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:43.446315 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.447420 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.449197 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.449974 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.450083 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.450351 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.450447 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.450570 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":146,"threads_started":1}
I20260620 12:13:43.451208 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.445505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.24.141.3:39993)
0620 12:13:43.446182 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.446196 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.446332 (+ 136us) server_negotiation.cc:407] Connection header received
0620 12:13:43.446429 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.446433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.446614 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.446767 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:43.446969 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.447900 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.449588 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.450333 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.450431 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.450545 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.450960 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.451075 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":109,"threads_started":1}
I20260620 12:13:43.451167 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.443881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:43105 (local address 127.0.0.1:40382)
0620 12:13:43.444744 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:43.444804 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:43.444981 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:43.445699 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:43.445706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:43.445732 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:43.445904 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.445912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.447466 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.447470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.448973 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.448983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.450565 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.450570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.450705 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.450774 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 12:13:43.450848 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 12:13:43.450980 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":129,"thread_start_us":111,"threads_started":1}
I20260620 12:13:43.451522 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.444888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:39993 (local address 127.0.0.1:60538)
0620 12:13:43.446190 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:43.446217 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:43.446302 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:43.446750 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:43.446753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:43.446770 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:43.446861 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.446867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.448051 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.448055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.449451 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.449461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.450597 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.450600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.450743 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.450780 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 12:13:43.450848 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:13:43.451431 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":726,"thread_start_us":104,"threads_started":1}
I20260620 12:13:43.451968 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.444198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.24.141.2:43105)
0620 12:13:43.445146 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.445160 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.445188 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:13:43.445264 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.445270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.445533 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.445709 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:43.446058 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.447312 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.449232 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.450143 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.450285 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.450486 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.451683 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.451818 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":168,"threads_started":1}
I20260620 12:13:43.454958 25835 tablet_service.cc:1467] Processing CreateTablet for tablet 91b5212a74294392b40a4896c029ee50 (DEFAULT_TABLE table=test-table [id=83b121803f9340fe8db80cd0a0353b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:43.455317 25969 tablet_service.cc:1467] Processing CreateTablet for tablet 91b5212a74294392b40a4896c029ee50 (DEFAULT_TABLE table=test-table [id=83b121803f9340fe8db80cd0a0353b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:43.456431 25835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b5212a74294392b40a4896c029ee50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:43.456784 25969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b5212a74294392b40a4896c029ee50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:43.458585 26103 tablet_service.cc:1467] Processing CreateTablet for tablet 91b5212a74294392b40a4896c029ee50 (DEFAULT_TABLE table=test-table [id=83b121803f9340fe8db80cd0a0353b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:43.461514 26103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b5212a74294392b40a4896c029ee50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:43.466987 26186 tablet_bootstrap.cc:492] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb: Bootstrap starting.
I20260620 12:13:43.471150 26186 tablet_bootstrap.cc:654] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:43.471750 26187 tablet_bootstrap.cc:492] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11: Bootstrap starting.
I20260620 12:13:43.472276 26186 log.cc:824] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:43.472570 26188 tablet_bootstrap.cc:492] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2: Bootstrap starting.
I20260620 12:13:43.474721 26187 tablet_bootstrap.cc:654] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:43.474876 26188 tablet_bootstrap.cc:654] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:43.475422 26186 tablet_bootstrap.cc:492] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb: No bootstrap required, opened a new log
I20260620 12:13:43.475733 26186 ts_tablet_manager.cc:1397] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 12:13:43.475888 26188 log.cc:824] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:43.476472 26187 log.cc:824] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:43.478361 26187 tablet_bootstrap.cc:492] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11: No bootstrap required, opened a new log
I20260620 12:13:43.478361 26188 tablet_bootstrap.cc:492] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2: No bootstrap required, opened a new log
I20260620 12:13:43.478574 26188 ts_tablet_manager.cc:1397] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:13:43.478574 26187 ts_tablet_manager.cc:1397] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 12:13:43.479509 26186 raft_consensus.cc:348] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.479803 26186 raft_consensus.cc:374] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:43.479844 26186 raft_consensus.cc:729] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f51097f2694b3e89df61470be938eb, State: Initialized, Role: FOLLOWER
I20260620 12:13:43.480336 26186 consensus_queue.cc:260] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.481238 26035 heartbeater.cc:503] Master 127.24.141.62:38273 was elected leader, sending a full tablet report...
I20260620 12:13:43.483108 26186 ts_tablet_manager.cc:1428] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:13:43.483896 26187 raft_consensus.cc:348] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.483896 26188 raft_consensus.cc:348] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.484236 26187 raft_consensus.cc:374] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:43.484236 26188 raft_consensus.cc:374] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:43.484302 26188 raft_consensus.cc:729] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33829aea76348cb883fc1e2e54b3ff2, State: Initialized, Role: FOLLOWER
I20260620 12:13:43.484311 26187 raft_consensus.cc:729] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad084ad0469448f7bc8019cdeaffcd11, State: Initialized, Role: FOLLOWER
I20260620 12:13:43.484875 26187 consensus_queue.cc:260] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.484875 26188 consensus_queue.cc:260] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.486047 25901 heartbeater.cc:503] Master 127.24.141.62:38273 was elected leader, sending a full tablet report...
I20260620 12:13:43.486202 26187 ts_tablet_manager.cc:1428] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 12:13:43.486202 26188 ts_tablet_manager.cc:1428] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 12:13:43.486707 26169 heartbeater.cc:503] Master 127.24.141.62:38273 was elected leader, sending a full tablet report...
I20260620 12:13:43.552541 26194 raft_consensus.cc:482] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:43.552819 26194 raft_consensus.cc:504] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.554409 26194 leader_election.cc:288] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a33829aea76348cb883fc1e2e54b3ff2 (127.24.141.1:38349), 24f51097f2694b3e89df61470be938eb (127.24.141.2:43105)
I20260620 12:13:43.559923 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.554960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:33241 (local address 127.24.141.2:43105)
0620 12:13:43.555118 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.555123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.555378 (+ 255us) server_negotiation.cc:407] Connection header received
0620 12:13:43.555448 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.555451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.555502 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.555582 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:43.555806 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.556651 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.558437 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.559104 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.559240 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.559306 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.559715 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.559803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:13:43.559921 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.555064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:33043 (local address 127.24.141.1:38349)
0620 12:13:43.555199 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.555202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.555217 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:13:43.555279 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.555282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.555354 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.555454 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:43.555779 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.556624 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.558426 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.559102 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.559240 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.559306 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.559649 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.559740 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:13:43.559938 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.554736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:38349 (local address 127.24.141.3:33043)
0620 12:13:43.554897 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:43.554917 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:43.555029 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:43.555492 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:43.555494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:43.555508 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:43.555644 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.555652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.556765 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.556769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.558277 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.558283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.559268 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.559273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.559500 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.559522 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:13:43.559624 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 12:13:43.559745 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":221}
I20260620 12:13:43.559947 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.554866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:43105 (local address 127.24.141.3:33241)
0620 12:13:43.555273 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:43.555283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:43.555373 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:43.555606 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:43.555608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:43.555621 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:43.555706 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.555711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.556765 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.556769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.558262 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:43.558278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.559296 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.559299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:43.559469 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.559522 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 12:13:43.559688 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 12:13:43.559745 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":48,"thread_start_us":152,"threads_started":1}
I20260620 12:13:43.560911 25855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b5212a74294392b40a4896c029ee50" candidate_uuid: "ad084ad0469448f7bc8019cdeaffcd11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33829aea76348cb883fc1e2e54b3ff2" is_pre_election: true
I20260620 12:13:43.560914 25989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b5212a74294392b40a4896c029ee50" candidate_uuid: "ad084ad0469448f7bc8019cdeaffcd11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f51097f2694b3e89df61470be938eb" is_pre_election: true
I20260620 12:13:43.561506 25989 raft_consensus.cc:2454] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad084ad0469448f7bc8019cdeaffcd11 in term 0.
I20260620 12:13:43.562173 26059 leader_election.cc:302] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f51097f2694b3e89df61470be938eb, ad084ad0469448f7bc8019cdeaffcd11; no voters:
I20260620 12:13:43.562526 26194 raft_consensus.cc:2781] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:43.562662 26194 raft_consensus.cc:482] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:43.562716 26194 raft_consensus.cc:3037] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:43.562983 25855 raft_consensus.cc:2454] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad084ad0469448f7bc8019cdeaffcd11 in term 0.
I20260620 12:13:43.564085 26194 raft_consensus.cc:504] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.564533 26194 leader_election.cc:288] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [CANDIDATE]: Term 1 election: Requested vote from peers a33829aea76348cb883fc1e2e54b3ff2 (127.24.141.1:38349), 24f51097f2694b3e89df61470be938eb (127.24.141.2:43105)
I20260620 12:13:43.564975 25855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b5212a74294392b40a4896c029ee50" candidate_uuid: "ad084ad0469448f7bc8019cdeaffcd11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33829aea76348cb883fc1e2e54b3ff2"
I20260620 12:13:43.564983 25989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b5212a74294392b40a4896c029ee50" candidate_uuid: "ad084ad0469448f7bc8019cdeaffcd11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f51097f2694b3e89df61470be938eb"
I20260620 12:13:43.565157 25989 raft_consensus.cc:3037] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:43.565238 25855 raft_consensus.cc:3037] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:43.566309 25989 raft_consensus.cc:2454] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad084ad0469448f7bc8019cdeaffcd11 in term 1.
I20260620 12:13:43.566591 25855 raft_consensus.cc:2454] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad084ad0469448f7bc8019cdeaffcd11 in term 1.
I20260620 12:13:43.566737 26059 leader_election.cc:302] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f51097f2694b3e89df61470be938eb, ad084ad0469448f7bc8019cdeaffcd11; no voters:
I20260620 12:13:43.567013 26194 raft_consensus.cc:2781] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:43.567232 26194 raft_consensus.cc:686] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [term 1 LEADER]: Becoming Leader. State: Replica: ad084ad0469448f7bc8019cdeaffcd11, State: Running, Role: LEADER
I20260620 12:13:43.567660 26194 consensus_queue.cc:237] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } }
I20260620 12:13:43.571724 25708 catalog_manager.cc:5495] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad084ad0469448f7bc8019cdeaffcd11 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "ad084ad0469448f7bc8019cdeaffcd11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33829aea76348cb883fc1e2e54b3ff2" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24f51097f2694b3e89df61470be938eb" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad084ad0469448f7bc8019cdeaffcd11" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 39993 } health_report { overall_health: HEALTHY } } }
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:13:43.586400 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.577176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.24.141.1:38349)
0620 12:13:43.577329 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.577334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.577406 (+ 72us) server_negotiation.cc:407] Connection header received
0620 12:13:43.577566 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.577570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.577626 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.577696 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:43.578470 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.579011 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.579868 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.580103 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.580194 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.582800 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:43.582812 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:43.582832 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:43.582859 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:43.584543 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.584951 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.584958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.584959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.585049 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.585332 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.585335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.585337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.585653 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:43.585878 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.586074 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.586208 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 12:13:43.600991 25902 tablet.cc:2369] T 91b5212a74294392b40a4896c029ee50 P a33829aea76348cb883fc1e2e54b3ff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 12:13:43.607739 26036 tablet.cc:2369] T 91b5212a74294392b40a4896c029ee50 P 24f51097f2694b3e89df61470be938eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:13:43.617892 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.608772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.24.141.1:38349)
0620 12:13:43.608916 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.608920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.609133 (+ 213us) server_negotiation.cc:407] Connection header received
0620 12:13:43.609288 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.609291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.609344 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.609478 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:43.610231 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.610816 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.611730 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.611962 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.612038 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.614646 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:43.614650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:43.614652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:43.614680 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:43.615971 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.616453 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.616456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.616457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.616537 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.616890 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.616893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.616894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.617127 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:43.617242 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.617572 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.617709 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 12:13:43.628037 26170 tablet.cc:2369] T 91b5212a74294392b40a4896c029ee50 P ad084ad0469448f7bc8019cdeaffcd11: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[25665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:13:43.647531 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:43.638558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.24.141.1:38349)
0620 12:13:43.638709 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:43.638714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:43.638902 (+ 188us) server_negotiation.cc:407] Connection header received
0620 12:13:43.639087 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:43.639090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:43.639140 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:43.639208 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:43.640047 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.640591 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.641542 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:43.641759 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:43.641847 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:43.644430 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:43.644433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:43.644436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:43.644467 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:43.645877 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.646288 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.646292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.646294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.646372 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:43.646622 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:43.646626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:43.646627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:43.646822 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:43.646949 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:43.647194 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 12:13:43.647325 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 12:13:43.649564 25815 tablet_service.cc:2908] Rejecting scan request for tablet 91b5212a74294392b40a4896c029ee50: Uninitialized: clean time has not yet been initialized
I20260620 12:13:43.652429 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 25772
I20260620 12:13:43.659065 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 25906
I20260620 12:13:43.664929 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 26040
I20260620 12:13:43.671857 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 25678
2026-06-20T12:13:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2497 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[26210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[26210](info): set up 2 sockets
Jun 20 12:13:43 dist-test-slave-rh5v krb5kdc[26210](info): commencing operation
krb5kdc: starting...
W20260620 12:13:44.637373 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:13:44 dist-test-slave-rh5v krb5kdc[26210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:13:44.772706 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44147
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:44147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:44.900266 26223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:44.900568 26223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:44.900611 26223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:44.904850 26223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:13:44.904932 26223 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:13:44.904948 26223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:44.904966 26223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:44.904981 26223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:13:44.904996 26223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:13:44.911096 26223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:44147
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:44.912358 26223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:44.913573 26223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:44.923583 26228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:44.923679 26229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:44.923619 26231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:44.924449 26223 server_base.cc:1034] running on GCE node
I20260620 12:13:44.925298 26223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:44.926888 26223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:44.928148 26223 hybrid_clock.cc:648] HybridClock initialized: now 1781957624928117 us; error 51 us; skew 500 ppm
Jun 20 12:13:44 dist-test-slave-rh5v krb5kdc[26210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957624, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:44.935456 26223 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:13:44.936619 26223 webserver.cc:460] Webserver started at http://127.24.141.62:35377/ using document root <none> and password file <none>
I20260620 12:13:44.937340 26223 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:44.937417 26223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:44.937640 26223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:44.939986 26223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "7bb2f78a12424336afd1aa289f252b72"
format_stamp: "Formatted at 2026-06-20 12:13:44 on dist-test-slave-rh5v"
server_key: "d96d37bdaf61875520fb0575a132f369"
server_key_iv: "63c9c6f3d155fac7388cff3120900da0"
server_key_version: "encryptionkey@0"
I20260620 12:13:44.940629 26223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "7bb2f78a12424336afd1aa289f252b72"
format_stamp: "Formatted at 2026-06-20 12:13:44 on dist-test-slave-rh5v"
server_key: "d96d37bdaf61875520fb0575a132f369"
server_key_iv: "63c9c6f3d155fac7388cff3120900da0"
server_key_version: "encryptionkey@0"
I20260620 12:13:44.945335 26223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 12:13:44.948359 26238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:44.950166 26223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:44.950367 26223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "7bb2f78a12424336afd1aa289f252b72"
format_stamp: "Formatted at 2026-06-20 12:13:44 on dist-test-slave-rh5v"
server_key: "d96d37bdaf61875520fb0575a132f369"
server_key_iv: "63c9c6f3d155fac7388cff3120900da0"
server_key_version: "encryptionkey@0"
I20260620 12:13:44.950507 26223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:44.963521 26223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:44.967064 26223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:44.967283 26223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:44.977437 26223 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:44147
I20260620 12:13:44.977460 26300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:44147 every 8 connection(s)
I20260620 12:13:44.978704 26223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:13:44.980998 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 26223
I20260620 12:13:44.981156 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:13:44.981480 25140 external_mini_cluster.cc:1428] Setting key f3471d97854bad7f0ad12f5f8b18d943
I20260620 12:13:44.982527 26301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:44.989553 26301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72: Bootstrap starting.
Jun 20 12:13:44 dist-test-slave-rh5v krb5kdc[26210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:44.992832 26301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:44.994665 26301 log.cc:824] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:44.997234 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:44.983153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.24.141.62:44147)
0620 12:13:44.983760 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:44.983782 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:44.983833 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:13:44.984665 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:44.984688 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:44.985236 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:44.985637 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:44.986508 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:44.987628 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:44.988501 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:44.989045 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:44.989254 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:44.991871 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:44.991876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:44.991902 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:44.991938 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:44.993720 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:44.994161 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:44.994168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:44.994174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:44.994300 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:44.994567 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:44.994570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:44.994572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:44.995102 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:44.995297 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:44.995622 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 12:13:44.996154 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":194,"threads_started":1}
I20260620 12:13:44.997685 26301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72: No bootstrap required, opened a new log
I20260620 12:13:45.001518 26301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2f78a12424336afd1aa289f252b72" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44147 } }
I20260620 12:13:45.001895 26301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:45.001996 26301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb2f78a12424336afd1aa289f252b72, State: Initialized, Role: FOLLOWER
I20260620 12:13:45.002512 26301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2f78a12424336afd1aa289f252b72" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44147 } }
I20260620 12:13:45.002664 26301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:13:45.002719 26301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:13:45.002810 26301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:45.003947 26301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2f78a12424336afd1aa289f252b72" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44147 } }
I20260620 12:13:45.004369 26301 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bb2f78a12424336afd1aa289f252b72; no voters:
I20260620 12:13:45.004715 26301 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:13:45.004876 26306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:45.005205 26306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb2f78a12424336afd1aa289f252b72, State: Running, Role: LEADER
I20260620 12:13:45.005621 26306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2f78a12424336afd1aa289f252b72" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44147 } }
I20260620 12:13:45.006407 26301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:13:45.007761 26308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb2f78a12424336afd1aa289f252b72. Latest consensus state: current_term: 1 leader_uuid: "7bb2f78a12424336afd1aa289f252b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2f78a12424336afd1aa289f252b72" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44147 } } }
I20260620 12:13:45.007761 26307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bb2f78a12424336afd1aa289f252b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2f78a12424336afd1aa289f252b72" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44147 } } }
I20260620 12:13:45.007944 26308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:45.007968 26307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:45.008801 26315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:13:45.013029 26315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:13:45.019621 26315 catalog_manager.cc:1347] Generated new cluster ID: e73fa11aebff4e228ff6467937e546bd
I20260620 12:13:45.019712 26315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:13:45.031498 26315 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:13:45.032840 26315 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:13:45.039217 26315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72: Generated new TSK 0
I20260620 12:13:45.040087 26315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:13:45.072764 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44147
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:45.200289 26327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:45.200656 26327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:45.200709 26327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:45.205127 26327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:45.205214 26327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:45.205258 26327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:13:45.205293 26327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:13:45.211467 26327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:45.212972 26327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:45.214417 26327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:45.226399 26333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:45.226373 26332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:45.226398 26335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:45.226928 26327 server_base.cc:1034] running on GCE node
I20260620 12:13:45.227537 26327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:45.228382 26327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:45.229626 26327 hybrid_clock.cc:648] HybridClock initialized: now 1781957625229586 us; error 54 us; skew 500 ppm
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:45.233776 26327 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:13:45.235212 26327 webserver.cc:460] Webserver started at http://127.24.141.1:45029/ using document root <none> and password file <none>
I20260620 12:13:45.235952 26327 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:45.236037 26327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:45.236269 26327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:45.238847 26327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "3b4efb4f90ba4d14918c01037ea66932"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "d8b2f5d98680410bf914c898f28c73ff"
server_key_iv: "69e8140f06c60eca546263a79f249893"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.239537 26327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "3b4efb4f90ba4d14918c01037ea66932"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "d8b2f5d98680410bf914c898f28c73ff"
server_key_iv: "69e8140f06c60eca546263a79f249893"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.244552 26327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:13:45.248262 26342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:45.250221 26327 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:13:45.250420 26327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "3b4efb4f90ba4d14918c01037ea66932"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "d8b2f5d98680410bf914c898f28c73ff"
server_key_iv: "69e8140f06c60eca546263a79f249893"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.250560 26327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:45.276081 26327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:45.280157 26327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:45.280385 26327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:45.282447 26327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:45.282521 26327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:45.282581 26327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:45.282610 26327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:45.294955 26327 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:43441
I20260620 12:13:45.294973 26457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:43441 every 8 connection(s)
I20260620 12:13:45.296267 26327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:13:45.300007 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.283676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:38683 (local address 127.24.141.62:44147)
0620 12:13:45.283896 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.283903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.285034 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 12:13:45.286438 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.286442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.286519 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.286597 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:45.288435 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.289366 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.291133 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.291535 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.291715 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.295656 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:45.295661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:45.295664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:45.295703 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:45.297602 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.298312 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.298316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.298318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.298420 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.298769 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.298772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.298774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.298993 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:45.299085 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.299669 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.299827 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 12:13:45.301121 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.284198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44147 (local address 127.24.141.1:38683)
0620 12:13:45.284815 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.284885 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.286124 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.286846 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.286855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.287453 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:45.288190 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.288208 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.289546 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.289549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.290821 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.290834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.291786 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.291790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.291998 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.292925 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:45.292956 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:45.295382 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:45.297770 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.297777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.297808 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.298190 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.298515 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.298518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.298521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.298667 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.299100 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:45.299108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:45.299412 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.299606 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.300049 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":171,"threads_started":1}
I20260620 12:13:45.301578 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 26327
I20260620 12:13:45.301729 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:13:45.302122 25140 external_mini_cluster.cc:1428] Setting key f298dff3acaa6b21d33ee2b2d8a659d5
I20260620 12:13:45.303161 26458 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44147
I20260620 12:13:45.303411 26458 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:45.304191 26458 heartbeater.cc:511] Master 127.24.141.62:44147 requested a full tablet report, sending...
I20260620 12:13:45.307936 26254 ts_manager.cc:194] Registered new tserver with Master: 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441)
I20260620 12:13:45.309599 26254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:38683
I20260620 12:13:45.315598 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.306282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44147 (local address 127.24.141.1:39809)
0620 12:13:45.306812 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.306826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.306944 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.307291 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.307294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.307653 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:45.307920 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.307953 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.308908 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.308911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.309757 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.309782 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.310394 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.310397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.310508 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.311228 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:45.311248 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:45.311754 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:45.313947 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.313951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.313954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.314220 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.314503 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.314506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.314508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.314592 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.314984 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:45.314988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:45.315112 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.315203 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.315392 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":259,"threads_started":1}
I20260620 12:13:45.315587 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.306356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39809 (local address 127.24.141.62:44147)
0620 12:13:45.306475 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.306479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.306908 (+ 429us) server_negotiation.cc:407] Connection header received
0620 12:13:45.307029 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.307032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.307084 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.307270 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:45.308099 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.308757 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.309993 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.310263 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.310365 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.311936 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:45.311940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:45.311943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:45.311969 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:45.313772 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.314324 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.314328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.314329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.314410 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.314677 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.314681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.314683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.314891 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:45.314981 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.315267 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.315405 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:13:45.337380 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44147
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:45.472608 26467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:45.472968 26467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:45.473022 26467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:45.477620 26467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:45.477720 26467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:45.477764 26467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:13:45.477807 26467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:13:45.484769 26467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:45.486356 26467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:45.487681 26467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:45.500207 26472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:45.500511 26467 server_base.cc:1034] running on GCE node
W20260620 12:13:45.500214 26475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:45.500205 26473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:45.501448 26467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:45.502388 26467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:45.503611 26467 hybrid_clock.cc:648] HybridClock initialized: now 1781957625503582 us; error 46 us; skew 500 ppm
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:45.507409 26467 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:13:45.508741 26467 webserver.cc:460] Webserver started at http://127.24.141.2:36563/ using document root <none> and password file <none>
I20260620 12:13:45.509508 26467 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:45.509581 26467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:45.509831 26467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:45.512388 26467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "dcae47d24b544799ac04a82cfd70ebd7"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "f2f5421b2b0a90c0b53d6117caa95f25"
server_key_iv: "3bdcdb0fd3a545ee88f44bcd01572cf1"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.513048 26467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "dcae47d24b544799ac04a82cfd70ebd7"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "f2f5421b2b0a90c0b53d6117caa95f25"
server_key_iv: "3bdcdb0fd3a545ee88f44bcd01572cf1"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.518193 26467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:13:45.521919 26482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:45.523960 26467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:13:45.524173 26467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "dcae47d24b544799ac04a82cfd70ebd7"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "f2f5421b2b0a90c0b53d6117caa95f25"
server_key_iv: "3bdcdb0fd3a545ee88f44bcd01572cf1"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.524350 26467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:45.542385 26467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:45.546175 26467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:45.546377 26467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:45.548137 26467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:45.548193 26467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:45.548242 26467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:45.548272 26467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:45.560343 26467 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:33507
I20260620 12:13:45.560462 26597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:33507 every 8 connection(s)
I20260620 12:13:45.561633 26467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:13:45.565588 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.549301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:38501 (local address 127.24.141.62:44147)
0620 12:13:45.549465 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.549471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.551008 (+ 1537us) server_negotiation.cc:407] Connection header received
0620 12:13:45.552466 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.552470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.552539 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.552625 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:45.554908 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.555676 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.557355 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.557602 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.557699 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.561307 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:45.561311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:45.561313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:45.561353 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:45.563063 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.563778 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.563782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.563783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.563909 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.564293 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.564295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.564296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.564516 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:45.564627 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.565297 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.565447 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:13:45.566084 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 26467
I20260620 12:13:45.566200 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:13:45.566517 25140 external_mini_cluster.cc:1428] Setting key d8df68310120baea9f174b3de083750f
I20260620 12:13:45.566720 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.550070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44147 (local address 127.24.141.2:38501)
0620 12:13:45.550792 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.550858 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.552195 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.552974 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.552988 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.553654 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:45.554597 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.554623 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.555883 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.555889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.557181 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.557204 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.557775 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.557780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.557997 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.558753 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:45.558773 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:45.560959 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:45.563206 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.563212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.563226 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.563590 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.564026 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.564029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.564030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.564189 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.564626 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:45.564632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:45.564938 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.565166 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.565542 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":193,"threads_started":1}
I20260620 12:13:45.568564 26598 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44147
I20260620 12:13:45.568763 26598 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:45.569609 26598 heartbeater.cc:511] Master 127.24.141.62:44147 requested a full tablet report, sending...
I20260620 12:13:45.572018 26254 ts_manager.cc:194] Registered new tserver with Master: dcae47d24b544799ac04a82cfd70ebd7 (127.24.141.2:33507)
I20260620 12:13:45.573028 26254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:38501
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:13:45.582439 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.571881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44147 (local address 127.24.141.2:45341)
0620 12:13:45.572213 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.572226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.572334 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.572794 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.572797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.573019 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:45.573199 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.573208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.574222 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.574226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.575358 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.575385 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.576327 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.576332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.576437 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.577242 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:45.577255 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:45.577659 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:45.580260 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.580264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.580266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.580643 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.581015 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.581018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.581020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.581179 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.581813 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:45.581817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:45.581942 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.582041 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.582204 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":127,"threads_started":1}
I20260620 12:13:45.582489 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.571957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:45341 (local address 127.24.141.62:44147)
0620 12:13:45.572102 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.572106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.572313 (+ 207us) server_negotiation.cc:407] Connection header received
0620 12:13:45.572521 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.572531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.572622 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.572742 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:45.573346 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.574009 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.575616 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.575998 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.576097 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.577895 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:45.577900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:45.577904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:45.577952 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:45.580053 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.580772 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.580775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.580777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.580870 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.581365 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.581370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.581372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.581671 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:45.581773 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.582116 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.582287 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:13:45.600914 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44147
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:45.733537 26607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:45.733860 26607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:45.733908 26607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:45.738198 26607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:45.738286 26607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:45.738325 26607 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:13:45.738360 26607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:13:45.744546 26607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:44147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:45.745906 26607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:45.747047 26607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:45.758522 26613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:45.758514 26615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:45.758522 26612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:45.759006 26607 server_base.cc:1034] running on GCE node
I20260620 12:13:45.759454 26607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:45.760151 26607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:45.761322 26607 hybrid_clock.cc:648] HybridClock initialized: now 1781957625761286 us; error 53 us; skew 500 ppm
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:45.764803 26607 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:13:45.766199 26607 webserver.cc:460] Webserver started at http://127.24.141.3:46379/ using document root <none> and password file <none>
I20260620 12:13:45.766943 26607 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:45.767019 26607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:45.767243 26607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:45.770154 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "216c70b6c6dc4c3196e093994b03a9b6"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "dc1fcb343c234d4ee6cc2ec90e050418"
server_key_iv: "c8076b014927ad926a2e7e7448326bb5"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.770833 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "216c70b6c6dc4c3196e093994b03a9b6"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "dc1fcb343c234d4ee6cc2ec90e050418"
server_key_iv: "c8076b014927ad926a2e7e7448326bb5"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.775985 26607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:13:45.779403 26622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:45.781309 26607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:13:45.781473 26607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "216c70b6c6dc4c3196e093994b03a9b6"
format_stamp: "Formatted at 2026-06-20 12:13:45 on dist-test-slave-rh5v"
server_key: "dc1fcb343c234d4ee6cc2ec90e050418"
server_key_iv: "c8076b014927ad926a2e7e7448326bb5"
server_key_version: "encryptionkey@0"
I20260620 12:13:45.781610 26607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:45.798766 26607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:45.802970 26607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:45.803167 26607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:45.805095 26607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:45.805156 26607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:45.805205 26607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:45.805222 26607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:45.817854 26607 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:46615
I20260620 12:13:45.817967 26737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:46615 every 8 connection(s)
I20260620 12:13:45.818974 26607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:13:45.819572 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 26607
I20260620 12:13:45.819716 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:13:45.820014 25140 external_mini_cluster.cc:1428] Setting key f635e11e16096764cce604e3242f2e32
I20260620 12:13:45.822615 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.806473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:55299 (local address 127.24.141.62:44147)
0620 12:13:45.806707 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.806713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.807971 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 12:13:45.809642 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.809646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.809715 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.809831 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:45.811383 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.811939 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.813648 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.813926 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.814040 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.817788 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:45.817793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:45.817797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:45.817859 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:45.819911 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.820554 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.820558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.820561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.820649 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.821096 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.821100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.821101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.821359 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:45.821472 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.822215 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.822372 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 12:13:45.823643 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.807025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44147 (local address 127.24.141.3:55299)
0620 12:13:45.807712 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.807796 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.809319 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.810110 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.810120 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.810552 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:45.811182 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.811200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.812076 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.812080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.813503 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.813517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.814137 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.814145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.814434 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.815163 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:45.815190 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:45.817483 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:45.820054 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.820060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.820077 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.820430 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.820766 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.820769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.820771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.820957 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.821482 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:45.821489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:45.821907 (+ 418us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.822112 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.822513 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":170,"threads_started":1}
I20260620 12:13:45.825459 26738 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44147
I20260620 12:13:45.825647 26738 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:45.826320 26738 heartbeater.cc:511] Master 127.24.141.62:44147 requested a full tablet report, sending...
I20260620 12:13:45.827857 26254 ts_manager.cc:194] Registered new tserver with Master: 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615)
I20260620 12:13:45.828701 26254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:55299
I20260620 12:13:45.836550 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:13:45.838627 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.828110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44147 (local address 127.24.141.3:57135)
0620 12:13:45.828479 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.828493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.828593 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.828933 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.828936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.829213 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:45.829363 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.829371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.830193 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.830197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.831001 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.831011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.832235 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.832240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.832384 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.833233 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:45.833251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:45.834125 (+ 874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:45.836764 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.836769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.836772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.837136 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.837466 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:45.837470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:45.837472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:45.837561 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:45.838027 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:45.838031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:45.838137 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.838221 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.838407 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":156,"threads_started":1}
I20260620 12:13:45.838614 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.828221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:57135 (local address 127.24.141.62:44147)
0620 12:13:45.828366 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.828372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.828575 (+ 203us) server_negotiation.cc:407] Connection header received
0620 12:13:45.828675 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.828678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.828734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.828885 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:45.829493 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.830045 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.831423 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.831786 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.832034 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.834428 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:45.834435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:45.834439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:45.834518 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:45.836597 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.837232 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.837236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.837254 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.837365 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.837663 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.837667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.837669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.837931 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:45.838027 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.838302 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.838437 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:13:45 dist-test-slave-rh5v krb5kdc[26210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:45.866956 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.858015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.24.141.62:44147)
0620 12:13:45.858158 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.858162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.858179 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:13:45.858237 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.858240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.858293 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.858381 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:45.859139 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.859724 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.860555 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.860778 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.860869 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.863374 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:45.863377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:45.863380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:45.863409 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:45.864919 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.865530 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.865533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.865534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.865623 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:45.865914 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:45.865917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:45.865919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:45.866147 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:45.866274 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.866615 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.866772 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:13:45.871241 26254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:13:45.874631 26254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:45.892784 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.886819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:46615 (local address 127.0.0.1:33966)
0620 12:13:45.887096 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.887112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.887199 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.888442 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.888445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.888465 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:45.888564 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.888569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.889876 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.889880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.890997 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.891007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.892388 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.892391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.892479 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.892535 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.892603 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.892671 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":79,"thread_start_us":113,"threads_started":1}
I20260620 12:13:45.892984 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.887392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.24.141.3:46615)
0620 12:13:45.887893 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.887906 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.887935 (+ 29us) server_negotiation.cc:407] Connection header received
0620 12:13:45.888014 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.888019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.888256 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.888455 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:45.888682 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.889739 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.891136 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.892046 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.892158 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.892346 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.892669 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.892801 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":144,"threads_started":1}
I20260620 12:13:45.892972 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.886574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.24.141.1:43441)
0620 12:13:45.886978 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.886986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.887013 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:13:45.887099 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.887105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.887379 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.887536 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:45.887892 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.889031 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.891202 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.892046 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.892158 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.892331 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.892684 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.892801 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":107,"threads_started":1}
I20260620 12:13:45.893395 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.885962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.24.141.2:33507)
0620 12:13:45.887268 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:45.887290 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:45.887318 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:13:45.887421 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:45.887426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:45.887676 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:45.887847 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:45.888122 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.889031 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.891261 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.891969 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.892080 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.892286 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:45.893012 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 12:13:45.893230 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":156,"threads_started":1}
I20260620 12:13:45.892784 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.885791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33507 (local address 127.0.0.1:46990)
0620 12:13:45.886419 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.886499 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.886723 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.887835 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.887839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.887860 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:45.888009 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.888017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.889151 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.889155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.891052 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.891066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.892115 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.892119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.892250 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.892294 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.892418 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.892586 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":169,"thread_start_us":141,"threads_started":1}
I20260620 12:13:45.893464 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:45.886123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43441 (local address 127.0.0.1:49086)
0620 12:13:45.886674 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:45.886691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:45.886798 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:45.887540 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:45.887546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:45.887569 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:45.887734 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.887741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.889203 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.889205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.891049 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:45.891066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:45.892412 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:45.892415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:45.892490 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:45.892552 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 12:13:45.893184 (+ 632us) client_negotiation.cc:240] Negotiation successful
0620 12:13:45.893291 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":189,"thread_start_us":109,"threads_started":1}
I20260620 12:13:45.897567 26672 tablet_service.cc:1467] Processing CreateTablet for tablet e1c5d2ab935e4eef92b7e5fb0839ca19 (DEFAULT_TABLE table=test-table [id=3154cca0fbba4665970ec0bbdab26cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:45.897610 26532 tablet_service.cc:1467] Processing CreateTablet for tablet e1c5d2ab935e4eef92b7e5fb0839ca19 (DEFAULT_TABLE table=test-table [id=3154cca0fbba4665970ec0bbdab26cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:45.897606 26392 tablet_service.cc:1467] Processing CreateTablet for tablet e1c5d2ab935e4eef92b7e5fb0839ca19 (DEFAULT_TABLE table=test-table [id=3154cca0fbba4665970ec0bbdab26cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:45.898742 26672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c5d2ab935e4eef92b7e5fb0839ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:45.898907 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c5d2ab935e4eef92b7e5fb0839ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:45.898907 26532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c5d2ab935e4eef92b7e5fb0839ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:45.908812 26762 tablet_bootstrap.cc:492] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Bootstrap starting.
I20260620 12:13:45.907866 26763 tablet_bootstrap.cc:492] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Bootstrap starting.
I20260620 12:13:45.910923 26761 tablet_bootstrap.cc:492] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Bootstrap starting.
I20260620 12:13:45.911311 26762 tablet_bootstrap.cc:654] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:45.912251 26762 log.cc:824] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:45.914001 26761 tablet_bootstrap.cc:654] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:45.914577 26762 tablet_bootstrap.cc:492] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: No bootstrap required, opened a new log
I20260620 12:13:45.914824 26762 ts_tablet_manager.cc:1397] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:13:45.915043 26761 log.cc:824] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:45.915311 26763 tablet_bootstrap.cc:654] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:45.916417 26763 log.cc:824] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:45.917552 26761 tablet_bootstrap.cc:492] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: No bootstrap required, opened a new log
I20260620 12:13:45.917841 26761 ts_tablet_manager.cc:1397] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 12:13:45.919111 26763 tablet_bootstrap.cc:492] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: No bootstrap required, opened a new log
I20260620 12:13:45.919360 26763 ts_tablet_manager.cc:1397] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 12:13:45.919227 26762 raft_consensus.cc:348] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:45.919557 26762 raft_consensus.cc:374] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:45.919623 26762 raft_consensus.cc:729] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcae47d24b544799ac04a82cfd70ebd7, State: Initialized, Role: FOLLOWER
I20260620 12:13:45.920192 26762 consensus_queue.cc:260] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:45.921203 26598 heartbeater.cc:503] Master 127.24.141.62:44147 was elected leader, sending a full tablet report...
I20260620 12:13:45.922945 26762 ts_tablet_manager.cc:1428] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260620 12:13:45.922660 26761 raft_consensus.cc:348] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:45.923043 26761 raft_consensus.cc:374] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:45.923103 26761 raft_consensus.cc:729] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4efb4f90ba4d14918c01037ea66932, State: Initialized, Role: FOLLOWER
I20260620 12:13:45.923707 26761 consensus_queue.cc:260] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:45.923825 26763 raft_consensus.cc:348] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:45.924095 26763 raft_consensus.cc:374] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:45.924163 26763 raft_consensus.cc:729] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216c70b6c6dc4c3196e093994b03a9b6, State: Initialized, Role: FOLLOWER
I20260620 12:13:45.924607 26458 heartbeater.cc:503] Master 127.24.141.62:44147 was elected leader, sending a full tablet report...
I20260620 12:13:45.924701 26763 consensus_queue.cc:260] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:45.925132 26761 ts_tablet_manager.cc:1428] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 12:13:45.925843 26763 ts_tablet_manager.cc:1428] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:13:45.926199 26738 heartbeater.cc:503] Master 127.24.141.62:44147 was elected leader, sending a full tablet report...
W20260620 12:13:46.048913 26459 tablet.cc:2369] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:13:46.055236 26767 raft_consensus.cc:482] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:46.055564 26767 raft_consensus.cc:504] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:46.057257 26767 leader_election.cc:288] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615), 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441)
I20260620 12:13:46.061721 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.057734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:37327 (local address 127.24.141.1:43441)
0620 12:13:46.057880 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.057885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.057997 (+ 112us) server_negotiation.cc:407] Connection header received
0620 12:13:46.058195 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.058198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.058251 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.058329 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:46.058647 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.059243 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.060424 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.061111 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.061227 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.061282 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.061515 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.061588 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:13:46.061756 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.057662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43441 (local address 127.24.141.2:37327)
0620 12:13:46.057879 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:46.057899 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:46.058055 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:46.058353 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:46.058356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:46.058375 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:46.058527 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.058536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.059394 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.059398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.060291 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.060302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.061242 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.061245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.061382 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.061412 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:13:46.061498 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:13:46.061588 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 12:13:46.062484 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.057672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:46615 (local address 127.24.141.2:42905)
0620 12:13:46.058060 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:46.058074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:46.058214 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:46.058502 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:46.058504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:46.058519 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:46.058663 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.058680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.059531 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.059534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.061040 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.061051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.062063 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.062066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.062165 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.062185 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:13:46.062253 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:13:46.062335 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":45,"thread_start_us":99,"threads_started":1}
I20260620 12:13:46.062484 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.057715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:42905 (local address 127.24.141.3:46615)
0620 12:13:46.057952 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.057957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.058198 (+ 241us) server_negotiation.cc:407] Connection header received
0620 12:13:46.058328 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.058331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.058389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.058475 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:46.058825 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.059405 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.061205 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.061806 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.061926 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.061982 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.062290 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.062371 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 12:13:46.062587 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c5d2ab935e4eef92b7e5fb0839ca19" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4efb4f90ba4d14918c01037ea66932" is_pre_election: true
I20260620 12:13:46.062999 26412 raft_consensus.cc:2454] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 0.
I20260620 12:13:46.063128 26692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c5d2ab935e4eef92b7e5fb0839ca19" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216c70b6c6dc4c3196e093994b03a9b6" is_pre_election: true
W20260620 12:13:46.063292 26599 tablet.cc:2369] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:13:46.063499 26692 raft_consensus.cc:2454] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 0.
I20260620 12:13:46.063587 26485 leader_election.cc:302] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4efb4f90ba4d14918c01037ea66932, dcae47d24b544799ac04a82cfd70ebd7; no voters:
I20260620 12:13:46.063953 26767 raft_consensus.cc:2781] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:46.064040 26767 raft_consensus.cc:482] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:46.064083 26767 raft_consensus.cc:3037] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.065337 26767 raft_consensus.cc:504] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
I20260620 12:13:46.065771 26767 leader_election.cc:288] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 election: Requested vote from peers 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615), 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441)
I20260620 12:13:46.066367 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c5d2ab935e4eef92b7e5fb0839ca19" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4efb4f90ba4d14918c01037ea66932"
I20260620 12:13:46.066399 26692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c5d2ab935e4eef92b7e5fb0839ca19" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216c70b6c6dc4c3196e093994b03a9b6"
I20260620 12:13:46.066565 26692 raft_consensus.cc:3037] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.066565 26412 raft_consensus.cc:3037] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.068389 26412 raft_consensus.cc:2454] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 1.
I20260620 12:13:46.068686 26692 raft_consensus.cc:2454] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 1.
I20260620 12:13:46.068943 26485 leader_election.cc:302] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4efb4f90ba4d14918c01037ea66932, dcae47d24b544799ac04a82cfd70ebd7; no voters:
I20260620 12:13:46.069357 26767 raft_consensus.cc:2781] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:46.069592 26767 raft_consensus.cc:686] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 LEADER]: Becoming Leader. State: Replica: dcae47d24b544799ac04a82cfd70ebd7, State: Running, Role: LEADER
I20260620 12:13:46.070019 26767 consensus_queue.cc:237] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } }
W20260620 12:13:46.071413 26739 tablet.cc:2369] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:13:46.074193 26253 catalog_manager.cc:5495] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcae47d24b544799ac04a82cfd70ebd7 (127.24.141.2). New cstate: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } health_report { overall_health: UNKNOWN } } }
I20260620 12:13:46.099615 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.095309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.24.141.2:33507)
0620 12:13:46.095572 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.095577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.095599 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:13:46.095786 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.095790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.095854 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.095970 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:46.096318 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.096860 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.097877 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.098124 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.098240 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.098566 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:46.099170 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:46.099277 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.099379 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.099449 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260620 12:13:46.107892 26692 raft_consensus.cc:1261] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Refusing update from remote peer dcae47d24b544799ac04a82cfd70ebd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:46.108925 26767 consensus_queue.cc:1035] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:46.109930 26412 raft_consensus.cc:1261] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Refusing update from remote peer dcae47d24b544799ac04a82cfd70ebd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:46.110664 26767 consensus_queue.cc:1035] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:46.120697 26775 mvcc.cc:205] Tried to move back new op lower bound from 7298898436526067712 to 7298898436386050048. Current Snapshot: MvccSnapshot[applied={T|T < 7298898436526067712}]
I20260620 12:13:46.120733 26776 mvcc.cc:205] Tried to move back new op lower bound from 7298898436526067712 to 7298898436386050048. Current Snapshot: MvccSnapshot[applied={T|T < 7298898436526067712}]
I20260620 12:13:46.135705 26255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:46.135973 26255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:46.138875 26255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72: Sending DeleteTablet for 3 replicas of tablet e1c5d2ab935e4eef92b7e5fb0839ca19
I20260620 12:13:46.139986 26392 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c5d2ab935e4eef92b7e5fb0839ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:46 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:49086
I20260620 12:13:46.139987 26532 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c5d2ab935e4eef92b7e5fb0839ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:46 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:46990
I20260620 12:13:46.140110 26672 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c5d2ab935e4eef92b7e5fb0839ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:46 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:33966
I20260620 12:13:46.140636 26784 tablet_replica.cc:330] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: stopping tablet replica
I20260620 12:13:46.141013 26786 tablet_replica.cc:330] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: stopping tablet replica
I20260620 12:13:46.141094 26784 raft_consensus.cc:2229] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:46.141503 26784 raft_consensus.cc:2258] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:46.141619 26786 raft_consensus.cc:2229] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:46.141983 26785 tablet_replica.cc:330] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: stopping tablet replica
I20260620 12:13:46.142132 26255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:13:46.142344 26785 raft_consensus.cc:2229] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:13:46.142618 26786 raft_consensus.cc:2258] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 12:13:46.142632 26255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:46.142979 26785 raft_consensus.cc:2258] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:46.144420 26784 ts_tablet_manager.cc:1905] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:46.145007 26785 ts_tablet_manager.cc:1905] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:46.146611 26786 ts_tablet_manager.cc:1905] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:46.148324 26785 ts_tablet_manager.cc:1918] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:13:46.148469 26785 log.cc:1196] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/e1c5d2ab935e4eef92b7e5fb0839ca19
I20260620 12:13:46.148854 26785 ts_tablet_manager.cc:1939] T e1c5d2ab935e4eef92b7e5fb0839ca19 P dcae47d24b544799ac04a82cfd70ebd7: Deleting consensus metadata
I20260620 12:13:46.149559 26532 tablet_service.cc:1467] Processing CreateTablet for tablet 175ebce7f6cf454fba4e4208657e9bbd (DEFAULT_TABLE table=test-table [id=e50f9513b8bc4e089f61d2d6fff2279b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:46.149863 26532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175ebce7f6cf454fba4e4208657e9bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:46.151368 26784 ts_tablet_manager.cc:1918] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:13:46.151502 26784 log.cc:1196] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/e1c5d2ab935e4eef92b7e5fb0839ca19
I20260620 12:13:46.151834 26784 ts_tablet_manager.cc:1939] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 3b4efb4f90ba4d14918c01037ea66932: Deleting consensus metadata
I20260620 12:13:46.152729 26762 tablet_bootstrap.cc:492] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: Bootstrap starting.
I20260620 12:13:46.153493 26239 catalog_manager.cc:4841] TS dcae47d24b544799ac04a82cfd70ebd7 (127.24.141.2:33507): tablet e1c5d2ab935e4eef92b7e5fb0839ca19 (table test-table [id=3154cca0fbba4665970ec0bbdab26cd1]) successfully deleted
I20260620 12:13:46.153971 26241 catalog_manager.cc:4841] TS 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441): tablet e1c5d2ab935e4eef92b7e5fb0839ca19 (table test-table [id=3154cca0fbba4665970ec0bbdab26cd1]) successfully deleted
I20260620 12:13:46.154074 26762 tablet_bootstrap.cc:654] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:46.154420 26392 tablet_service.cc:1467] Processing CreateTablet for tablet 175ebce7f6cf454fba4e4208657e9bbd (DEFAULT_TABLE table=test-table [id=e50f9513b8bc4e089f61d2d6fff2279b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:46.154783 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175ebce7f6cf454fba4e4208657e9bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:46.156116 26786 ts_tablet_manager.cc:1918] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:13:46.156297 26786 log.cc:1196] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/e1c5d2ab935e4eef92b7e5fb0839ca19
I20260620 12:13:46.156636 26786 ts_tablet_manager.cc:1939] T e1c5d2ab935e4eef92b7e5fb0839ca19 P 216c70b6c6dc4c3196e093994b03a9b6: Deleting consensus metadata
I20260620 12:13:46.158149 26672 tablet_service.cc:1467] Processing CreateTablet for tablet 175ebce7f6cf454fba4e4208657e9bbd (DEFAULT_TABLE table=test-table [id=e50f9513b8bc4e089f61d2d6fff2279b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:46.158286 26761 tablet_bootstrap.cc:492] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: Bootstrap starting.
I20260620 12:13:46.158454 26672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175ebce7f6cf454fba4e4208657e9bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:46.159457 26761 tablet_bootstrap.cc:654] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:46.160888 26762 tablet_bootstrap.cc:492] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: No bootstrap required, opened a new log
I20260620 12:13:46.160969 26761 tablet_bootstrap.cc:492] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: No bootstrap required, opened a new log
I20260620 12:13:46.161005 26762 ts_tablet_manager.cc:1397] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260620 12:13:46.161037 26761 ts_tablet_manager.cc:1397] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:46.161464 26242 catalog_manager.cc:4841] TS 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615): tablet e1c5d2ab935e4eef92b7e5fb0839ca19 (table test-table [id=3154cca0fbba4665970ec0bbdab26cd1]) successfully deleted
I20260620 12:13:46.162108 26763 tablet_bootstrap.cc:492] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: Bootstrap starting.
I20260620 12:13:46.162070 26762 raft_consensus.cc:348] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.162212 26762 raft_consensus.cc:374] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:46.162251 26762 raft_consensus.cc:729] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcae47d24b544799ac04a82cfd70ebd7, State: Initialized, Role: FOLLOWER
I20260620 12:13:46.162415 26762 consensus_queue.cc:260] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.162940 26761 raft_consensus.cc:348] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.163059 26761 raft_consensus.cc:374] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:46.163089 26761 raft_consensus.cc:729] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4efb4f90ba4d14918c01037ea66932, State: Initialized, Role: FOLLOWER
I20260620 12:13:46.163307 26763 tablet_bootstrap.cc:654] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:46.163312 26762 ts_tablet_manager.cc:1428] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:13:46.164060 26761 consensus_queue.cc:260] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.165182 26763 tablet_bootstrap.cc:492] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: No bootstrap required, opened a new log
I20260620 12:13:46.165282 26763 ts_tablet_manager.cc:1397] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:46.165791 26763 raft_consensus.cc:348] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.165925 26763 raft_consensus.cc:374] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:46.165954 26763 raft_consensus.cc:729] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216c70b6c6dc4c3196e093994b03a9b6, State: Initialized, Role: FOLLOWER
I20260620 12:13:46.166091 26763 consensus_queue.cc:260] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.166360 26763 ts_tablet_manager.cc:1428] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:13:46.166386 26761 ts_tablet_manager.cc:1428] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260620 12:13:46.378983 26767 raft_consensus.cc:482] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:46.379206 26767 raft_consensus.cc:504] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.379776 26767 leader_election.cc:288] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441), 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615)
I20260620 12:13:46.380429 26692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ebce7f6cf454fba4e4208657e9bbd" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216c70b6c6dc4c3196e093994b03a9b6" is_pre_election: true
I20260620 12:13:46.380415 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ebce7f6cf454fba4e4208657e9bbd" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4efb4f90ba4d14918c01037ea66932" is_pre_election: true
I20260620 12:13:46.380595 26692 raft_consensus.cc:2454] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 0.
I20260620 12:13:46.380606 26412 raft_consensus.cc:2454] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 0.
I20260620 12:13:46.380968 26486 leader_election.cc:302] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216c70b6c6dc4c3196e093994b03a9b6, dcae47d24b544799ac04a82cfd70ebd7; no voters:
I20260620 12:13:46.381248 26767 raft_consensus.cc:2781] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:46.381330 26767 raft_consensus.cc:482] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:46.381373 26767 raft_consensus.cc:3037] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.382586 26767 raft_consensus.cc:504] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.382980 26767 leader_election.cc:288] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 election: Requested vote from peers 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441), 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615)
I20260620 12:13:46.383424 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ebce7f6cf454fba4e4208657e9bbd" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4efb4f90ba4d14918c01037ea66932"
I20260620 12:13:46.383580 26412 raft_consensus.cc:3037] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.383594 26692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ebce7f6cf454fba4e4208657e9bbd" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216c70b6c6dc4c3196e093994b03a9b6"
I20260620 12:13:46.383728 26692 raft_consensus.cc:3037] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.384903 26692 raft_consensus.cc:2454] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 1.
I20260620 12:13:46.384904 26412 raft_consensus.cc:2454] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 1.
I20260620 12:13:46.385284 26486 leader_election.cc:302] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216c70b6c6dc4c3196e093994b03a9b6, dcae47d24b544799ac04a82cfd70ebd7; no voters:
I20260620 12:13:46.385520 26767 raft_consensus.cc:2781] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:46.385666 26767 raft_consensus.cc:686] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 1 LEADER]: Becoming Leader. State: Replica: dcae47d24b544799ac04a82cfd70ebd7, State: Running, Role: LEADER
I20260620 12:13:46.385860 26767 consensus_queue.cc:237] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.388016 26255 catalog_manager.cc:5495] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcae47d24b544799ac04a82cfd70ebd7 (127.24.141.2). New cstate: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } health_report { overall_health: UNKNOWN } } }
Jun 20 12:13:46 dist-test-slave-rh5v krb5kdc[26210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957624, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:46.437005 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.427385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44147 (local address 127.0.0.1:56366)
0620 12:13:46.427782 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:46.427799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:46.427973 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:46.428227 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:46.428230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:46.428755 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:46.428861 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.428867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.429731 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.429734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.430299 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.430306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.430748 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.430752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.430823 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.431470 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:46.431482 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:46.433197 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:46.435241 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:46.435249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:46.435255 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:46.435537 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:46.435901 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:46.435903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:46.435904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:46.436005 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:46.436399 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:46.436405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:46.436544 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 12:13:46.436644 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 12:13:46.436828 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":140,"threads_started":1}
I20260620 12:13:46.437166 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.427432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.24.141.62:44147)
0620 12:13:46.427771 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.427783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.427892 (+ 109us) server_negotiation.cc:407] Connection header received
0620 12:13:46.428065 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.428068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.428108 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.428193 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:46.428985 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.429633 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.430412 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.430641 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.430727 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.433463 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:46.433468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:46.433471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:46.433512 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:46.435083 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:46.435669 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:46.435673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:46.435675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:46.435794 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:46.436083 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:46.436085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:46.436086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:46.436292 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:46.436376 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.436707 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.436922 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260620 12:13:46.442515 26255 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:56366:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:13:46.443707 26255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:46.450690 26532 tablet_service.cc:1467] Processing CreateTablet for tablet f274b3ce6467421789cd60ca7ff385e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22e5d54c63804c8eaaa519876dfb3b8d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:13:46.450644 26672 tablet_service.cc:1467] Processing CreateTablet for tablet f274b3ce6467421789cd60ca7ff385e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22e5d54c63804c8eaaa519876dfb3b8d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:13:46.451246 26672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f274b3ce6467421789cd60ca7ff385e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:46.451274 26532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f274b3ce6467421789cd60ca7ff385e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:46.450644 26392 tablet_service.cc:1467] Processing CreateTablet for tablet f274b3ce6467421789cd60ca7ff385e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22e5d54c63804c8eaaa519876dfb3b8d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:13:46.453750 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f274b3ce6467421789cd60ca7ff385e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:46.454485 26762 tablet_bootstrap.cc:492] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: Bootstrap starting.
I20260620 12:13:46.454694 26763 tablet_bootstrap.cc:492] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: Bootstrap starting.
I20260620 12:13:46.455586 26762 tablet_bootstrap.cc:654] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:46.456161 26763 tablet_bootstrap.cc:654] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:46.457026 26762 tablet_bootstrap.cc:492] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: No bootstrap required, opened a new log
I20260620 12:13:46.457183 26762 ts_tablet_manager.cc:1397] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:46.457609 26762 raft_consensus.cc:348] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.457718 26762 raft_consensus.cc:374] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:46.457727 26763 tablet_bootstrap.cc:492] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: No bootstrap required, opened a new log
I20260620 12:13:46.457738 26762 raft_consensus.cc:729] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcae47d24b544799ac04a82cfd70ebd7, State: Initialized, Role: FOLLOWER
I20260620 12:13:46.457823 26763 ts_tablet_manager.cc:1397] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:46.457868 26762 consensus_queue.cc:260] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.458156 26762 ts_tablet_manager.cc:1428] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:13:46.458117 26767 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.458259 26767 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:46.459211 26763 raft_consensus.cc:348] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.459363 26763 raft_consensus.cc:374] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:46.459396 26763 raft_consensus.cc:729] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216c70b6c6dc4c3196e093994b03a9b6, State: Initialized, Role: FOLLOWER
I20260620 12:13:46.459642 26761 tablet_bootstrap.cc:492] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: Bootstrap starting.
I20260620 12:13:46.459704 26763 consensus_queue.cc:260] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.460078 26769 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.460246 26769 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:46.460520 26763 ts_tablet_manager.cc:1428] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:46.460840 26761 tablet_bootstrap.cc:654] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:46.462338 26761 tablet_bootstrap.cc:492] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: No bootstrap required, opened a new log
I20260620 12:13:46.462440 26761 ts_tablet_manager.cc:1397] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:46.462874 26761 raft_consensus.cc:348] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.463016 26761 raft_consensus.cc:374] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:46.463063 26761 raft_consensus.cc:729] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4efb4f90ba4d14918c01037ea66932, State: Initialized, Role: FOLLOWER
I20260620 12:13:46.463193 26761 consensus_queue.cc:260] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.463486 26761 ts_tablet_manager.cc:1428] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:13:46.463452 26768 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.463605 26768 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:46.705798 26767 raft_consensus.cc:482] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:46.705961 26767 raft_consensus.cc:504] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.706399 26767 leader_election.cc:288] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441), 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615)
I20260620 12:13:46.707079 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f274b3ce6467421789cd60ca7ff385e0" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4efb4f90ba4d14918c01037ea66932" is_pre_election: true
I20260620 12:13:46.707088 26692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f274b3ce6467421789cd60ca7ff385e0" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216c70b6c6dc4c3196e093994b03a9b6" is_pre_election: true
I20260620 12:13:46.707244 26412 raft_consensus.cc:2454] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 0.
I20260620 12:13:46.707346 26692 raft_consensus.cc:2454] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 0.
I20260620 12:13:46.707618 26485 leader_election.cc:302] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4efb4f90ba4d14918c01037ea66932, dcae47d24b544799ac04a82cfd70ebd7; no voters:
I20260620 12:13:46.707862 26767 raft_consensus.cc:2781] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:46.707947 26767 raft_consensus.cc:482] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:46.707976 26767 raft_consensus.cc:3037] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.709349 26767 raft_consensus.cc:504] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.709731 26767 leader_election.cc:288] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 election: Requested vote from peers 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441), 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615)
I20260620 12:13:46.710160 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f274b3ce6467421789cd60ca7ff385e0" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4efb4f90ba4d14918c01037ea66932"
I20260620 12:13:46.710373 26412 raft_consensus.cc:3037] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.710346 26692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f274b3ce6467421789cd60ca7ff385e0" candidate_uuid: "dcae47d24b544799ac04a82cfd70ebd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216c70b6c6dc4c3196e093994b03a9b6"
I20260620 12:13:46.710486 26692 raft_consensus.cc:3037] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:46.711676 26692 raft_consensus.cc:2454] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 1.
I20260620 12:13:46.711676 26412 raft_consensus.cc:2454] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcae47d24b544799ac04a82cfd70ebd7 in term 1.
I20260620 12:13:46.712042 26485 leader_election.cc:302] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4efb4f90ba4d14918c01037ea66932, dcae47d24b544799ac04a82cfd70ebd7; no voters:
I20260620 12:13:46.712232 26767 raft_consensus.cc:2781] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:46.712332 26767 raft_consensus.cc:686] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [term 1 LEADER]: Becoming Leader. State: Replica: dcae47d24b544799ac04a82cfd70ebd7, State: Running, Role: LEADER
I20260620 12:13:46.712486 26767 consensus_queue.cc:237] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } }
I20260620 12:13:46.713229 26779 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: TxnStatusTablet state changed. Reason: New leader dcae47d24b544799ac04a82cfd70ebd7. Latest consensus state: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.713347 26779 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:13:46.713699 26802 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:13:46.713936 26802 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:13:46.714413 26253 catalog_manager.cc:5495] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcae47d24b544799ac04a82cfd70ebd7 (127.24.141.2). New cstate: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } health_report { overall_health: UNKNOWN } } }
I20260620 12:13:46.765926 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.762173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33507 (local address 127.0.0.1:47004)
0620 12:13:46.762395 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:46.762415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:46.762561 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:46.763031 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:46.763034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:46.763042 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:46.763165 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.763170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.763951 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.763954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.764597 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.764606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.765052 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.765091 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.765173 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.765227 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:46.765589 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:46.765607 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:13:46.765665 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 12:13:46.765765 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 12:13:46.765928 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.762512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.24.141.2:33507)
0620 12:13:46.762777 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.762781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.762793 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:13:46.762845 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.762847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.762904 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.762987 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:46.763263 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.763838 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.764706 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.764943 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.765034 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.765416 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:46.765510 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:46.765572 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.765686 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.765744 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260620 12:13:46.769718 26412 raft_consensus.cc:1261] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Refusing update from remote peer dcae47d24b544799ac04a82cfd70ebd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:46.769737 26692 raft_consensus.cc:1261] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Refusing update from remote peer dcae47d24b544799ac04a82cfd70ebd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:46.770191 26779 consensus_queue.cc:1035] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:46.770521 26779 consensus_queue.cc:1035] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:46.771765 26768 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: TxnStatusTablet state changed. Reason: New leader dcae47d24b544799ac04a82cfd70ebd7. Latest consensus state: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.771894 26768 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:46.772246 26769 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: TxnStatusTablet state changed. Reason: New leader dcae47d24b544799ac04a82cfd70ebd7. Latest consensus state: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.772400 26769 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:46.774065 26767 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.774184 26767 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:13:46.774288 26806 mvcc.cc:205] Tried to move back new op lower bound from 7298898439243411456 to 7298898439015768064. Current Snapshot: MvccSnapshot[applied={T|T < 7298898439243411456}]
I20260620 12:13:46.775347 26769 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.775519 26769 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P 216c70b6c6dc4c3196e093994b03a9b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:46.777659 26779 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.777793 26779 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P dcae47d24b544799ac04a82cfd70ebd7: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:13:46.779093 26768 tablet_replica.cc:439] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcae47d24b544799ac04a82cfd70ebd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 } } peers { permanent_uuid: "dcae47d24b544799ac04a82cfd70ebd7" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33507 } } peers { permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 } } }
I20260620 12:13:46.779225 26768 tablet_replica.cc:442] T f274b3ce6467421789cd60ca7ff385e0 P 3b4efb4f90ba4d14918c01037ea66932: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:46.787113 26767 consensus_queue.cc:1035] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216c70b6c6dc4c3196e093994b03a9b6" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:46.791564 26779 consensus_queue.cc:1035] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4efb4f90ba4d14918c01037ea66932" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:46.798707 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.793916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:58663 (local address 127.24.141.2:33507)
0620 12:13:46.794062 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.794067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.794628 (+ 561us) server_negotiation.cc:407] Connection header received
0620 12:13:46.794778 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.794781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.794846 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.794927 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:46.795227 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.796061 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.797300 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.797656 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.797747 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.798128 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:46.798212 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:46.798264 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.798415 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.798477 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:13:46.798761 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.793669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33507 (local address 127.24.141.2:58663)
0620 12:13:46.794517 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:46.794534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:46.794666 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:46.795035 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:46.795038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:46.795045 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:46.795134 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.795140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.796220 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.796223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.797165 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:46.797177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.797798 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.797802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:46.797901 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.797986 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:46.798293 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:46.798299 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:13:46.798364 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:13:46.798470 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":116,"threads_started":1}
I20260620 12:13:46.861743 26255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:46.861986 26255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:46.864674 26255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bb2f78a12424336afd1aa289f252b72: Sending DeleteTablet for 3 replicas of tablet 175ebce7f6cf454fba4e4208657e9bbd
I20260620 12:13:46.865527 26532 tablet_service.cc:1514] Processing DeleteTablet for tablet 175ebce7f6cf454fba4e4208657e9bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:46 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:46990
I20260620 12:13:46.865523 26392 tablet_service.cc:1514] Processing DeleteTablet for tablet 175ebce7f6cf454fba4e4208657e9bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:46 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:49086
I20260620 12:13:46.865696 26672 tablet_service.cc:1514] Processing DeleteTablet for tablet 175ebce7f6cf454fba4e4208657e9bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:46 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:33966
I20260620 12:13:46.866056 26822 tablet_replica.cc:330] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: stopping tablet replica
I20260620 12:13:46.866225 26822 raft_consensus.cc:2229] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:46.866391 26822 raft_consensus.cc:2258] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:46.866261 26824 tablet_replica.cc:330] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: stopping tablet replica
I20260620 12:13:46.866842 26823 tablet_replica.cc:330] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: stopping tablet replica
I20260620 12:13:46.867038 26824 raft_consensus.cc:2229] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:13:46.867233 26823 raft_consensus.cc:2229] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:46.867246 26822 ts_tablet_manager.cc:1905] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:46.867568 26824 raft_consensus.cc:2258] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:46.867766 26823 raft_consensus.cc:2258] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:46.868757 26823 ts_tablet_manager.cc:1905] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:46.869259 26824 ts_tablet_manager.cc:1905] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:46.871253 26822 ts_tablet_manager.cc:1918] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:46.871371 26822 log.cc:1196] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/175ebce7f6cf454fba4e4208657e9bbd
I20260620 12:13:46.871655 26822 ts_tablet_manager.cc:1939] T 175ebce7f6cf454fba4e4208657e9bbd P 3b4efb4f90ba4d14918c01037ea66932: Deleting consensus metadata
I20260620 12:13:46.872648 26241 catalog_manager.cc:4841] TS 3b4efb4f90ba4d14918c01037ea66932 (127.24.141.1:43441): tablet 175ebce7f6cf454fba4e4208657e9bbd (table test-table [id=e50f9513b8bc4e089f61d2d6fff2279b]) successfully deleted
I20260620 12:13:46.872843 26823 ts_tablet_manager.cc:1918] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:46.872985 26823 log.cc:1196] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/175ebce7f6cf454fba4e4208657e9bbd
I20260620 12:13:46.873000 26824 ts_tablet_manager.cc:1918] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:46.873162 26824 log.cc:1196] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/175ebce7f6cf454fba4e4208657e9bbd
I20260620 12:13:46.873395 26823 ts_tablet_manager.cc:1939] T 175ebce7f6cf454fba4e4208657e9bbd P 216c70b6c6dc4c3196e093994b03a9b6: Deleting consensus metadata
I20260620 12:13:46.873485 26824 ts_tablet_manager.cc:1939] T 175ebce7f6cf454fba4e4208657e9bbd P dcae47d24b544799ac04a82cfd70ebd7: Deleting consensus metadata
I20260620 12:13:46.874187 26242 catalog_manager.cc:4841] TS 216c70b6c6dc4c3196e093994b03a9b6 (127.24.141.3:46615): tablet 175ebce7f6cf454fba4e4208657e9bbd (table test-table [id=e50f9513b8bc4e089f61d2d6fff2279b]) successfully deleted
I20260620 12:13:46.874460 26239 catalog_manager.cc:4841] TS dcae47d24b544799ac04a82cfd70ebd7 (127.24.141.2:33507): tablet 175ebce7f6cf454fba4e4208657e9bbd (table test-table [id=e50f9513b8bc4e089f61d2d6fff2279b]) successfully deleted
Jun 20 12:13:46 dist-test-slave-rh5v krb5kdc[26210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:13:46.884742 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.867785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.24.141.1:43441)
0620 12:13:46.873119 (+ 5334us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.873124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.874532 (+ 1408us) server_negotiation.cc:407] Connection header received
0620 12:13:46.874573 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.874576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.874637 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.874725 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:46.875478 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.876062 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.876879 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.877189 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.877285 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.880248 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:46.880262 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:46.880283 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:46.880314 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:46.882476 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:46.883164 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:46.883170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:46.883171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:46.883263 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:46.883588 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:46.883591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:46.883592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:46.883921 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:46.884114 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.884422 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.884560 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5214,"thread_start_us":116,"threads_started":1}
W20260620 12:13:46.885516 26452 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49102
I20260620 12:13:46.895386 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:46.887505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.24.141.62:44147)
0620 12:13:46.887673 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:46.887681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:46.887721 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:13:46.887885 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:46.887889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:46.887961 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:46.888032 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:46.888769 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.889467 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.890275 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:46.890508 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:46.890597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:46.892122 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:46.892126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:46.892128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:46.892154 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:46.893674 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:46.894129 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:46.894134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:46.894135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:46.894216 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:46.894475 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:46.894479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:46.894480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:46.894683 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:46.894805 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:46.895049 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 12:13:46.895197 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 12:13:46.895973 26255 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56380
I20260620 12:13:46.897511 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 26327
I20260620 12:13:46.904059 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 26467
I20260620 12:13:46.911960 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 26607
I20260620 12:13:46.918824 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 26223
2026-06-20T12:13:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3246 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:46 dist-test-slave-rh5v krb5kdc[26832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:46 dist-test-slave-rh5v krb5kdc[26832](info): set up 2 sockets
Jun 20 12:13:46 dist-test-slave-rh5v krb5kdc[26832](info): commencing operation
krb5kdc: starting...
W20260620 12:13:47.885454 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:13:47 dist-test-slave-rh5v krb5kdc[26832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:13:48.020864 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38009
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:38009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:48.150430 26845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:48.150743 26845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:48.150802 26845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:48.154964 26845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:13:48.155032 26845 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:13:48.155056 26845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:48.155081 26845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:48.155102 26845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:13:48.155130 26845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:13:48.160768 26845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:38009
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:48.162103 26845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:48.163322 26845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:13:48.173367 26845 server_base.cc:1034] running on GCE node
W20260620 12:13:48.173300 26850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:48.173302 26853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:48.173336 26851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:48.174356 26845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:48.175737 26845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:48.176923 26845 hybrid_clock.cc:648] HybridClock initialized: now 1781957628176889 us; error 49 us; skew 500 ppm
Jun 20 12:13:48 dist-test-slave-rh5v krb5kdc[26832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:48.179934 26845 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:13:48.181020 26845 webserver.cc:460] Webserver started at http://127.24.141.62:46623/ using document root <none> and password file <none>
I20260620 12:13:48.181720 26845 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:48.181789 26845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:48.181993 26845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:48.184378 26845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "6962a35802984f52ae28cbb04d4304d3"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "e24d79f3b318ad4c2fde95bbf87caadf"
server_key_iv: "015fa53634132078751170b33a30266b"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.184978 26845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "6962a35802984f52ae28cbb04d4304d3"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "e24d79f3b318ad4c2fde95bbf87caadf"
server_key_iv: "015fa53634132078751170b33a30266b"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.189720 26845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:13:48.192613 26860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:48.194305 26845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:48.194473 26845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "6962a35802984f52ae28cbb04d4304d3"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "e24d79f3b318ad4c2fde95bbf87caadf"
server_key_iv: "015fa53634132078751170b33a30266b"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.194621 26845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:48.207598 26845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:48.211377 26845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:48.211592 26845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:48.221148 26845 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:38009
I20260620 12:13:48.221235 26922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:38009 every 8 connection(s)
I20260620 12:13:48.222368 26845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:13:48.226224 26923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:48.229424 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 26845
I20260620 12:13:48.229575 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:13:48.229894 25140 external_mini_cluster.cc:1428] Setting key c86753d99932876605f4bf91d25680f5
I20260620 12:13:48.233856 26923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Bootstrap starting.
I20260620 12:13:48.236624 26923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:48.237594 26923 log.cc:824] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:48.240546 26923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: No bootstrap required, opened a new log
Jun 20 12:13:48 dist-test-slave-rh5v krb5kdc[26832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:48.245692 26923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962a35802984f52ae28cbb04d4304d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38009 } }
I20260620 12:13:48.246034 26923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:48.246086 26923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6962a35802984f52ae28cbb04d4304d3, State: Initialized, Role: FOLLOWER
I20260620 12:13:48.246681 26923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962a35802984f52ae28cbb04d4304d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38009 } }
I20260620 12:13:48.246841 26923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:13:48.246910 26923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:13:48.247035 26923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:48.248389 26923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962a35802984f52ae28cbb04d4304d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38009 } }
I20260620 12:13:48.248800 26923 leader_election.cc:302] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6962a35802984f52ae28cbb04d4304d3; no voters:
I20260620 12:13:48.249218 26923 leader_election.cc:288] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:13:48.249326 26928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:48.249524 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.231870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.24.141.62:38009)
0620 12:13:48.232641 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:48.232667 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:48.232723 (+ 56us) server_negotiation.cc:407] Connection header received
0620 12:13:48.233707 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:48.233736 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:48.234344 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:48.234855 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:48.235816 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.237028 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.238325 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.239010 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.239211 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.242637 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:48.242645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:48.242667 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:48.242704 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:48.245192 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.245910 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.245917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.245923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.246087 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.246557 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.246563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.246564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.247301 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:48.247546 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:48.247947 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 12:13:48.248367 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":190,"threads_started":1}
I20260620 12:13:48.249516 26928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [term 1 LEADER]: Becoming Leader. State: Replica: 6962a35802984f52ae28cbb04d4304d3, State: Running, Role: LEADER
I20260620 12:13:48.250396 26928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962a35802984f52ae28cbb04d4304d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38009 } }
I20260620 12:13:48.250898 26923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:13:48.252414 26929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6962a35802984f52ae28cbb04d4304d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962a35802984f52ae28cbb04d4304d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38009 } } }
I20260620 12:13:48.252619 26929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:48.252954 26930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6962a35802984f52ae28cbb04d4304d3. Latest consensus state: current_term: 1 leader_uuid: "6962a35802984f52ae28cbb04d4304d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962a35802984f52ae28cbb04d4304d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38009 } } }
I20260620 12:13:48.253083 26930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:48.253249 26939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:13:48.257661 26939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:13:48.266147 26939 catalog_manager.cc:1347] Generated new cluster ID: 75cfa5872d14412dbdae2964241d71b6
I20260620 12:13:48.266247 26939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:13:48.274802 26939 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:13:48.275997 26939 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:13:48.283819 26939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Generated new TSK 0
I20260620 12:13:48.284722 26939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:13:48.320654 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38009
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:48.486923 26949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:48.487293 26949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:48.487370 26949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:48.491868 26949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:48.491953 26949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:48.492017 26949 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:13:48.492044 26949 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:13:48.492096 26949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:13:48.499365 26949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:38009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:48.500888 26949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:48.502228 26949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:48.513777 26955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:48.513829 26954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:48.513993 26949 server_base.cc:1034] running on GCE node
W20260620 12:13:48.513779 26957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:48.514672 26949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:48.515636 26949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:48.516880 26949 hybrid_clock.cc:648] HybridClock initialized: now 1781957628516854 us; error 40 us; skew 500 ppm
Jun 20 12:13:48 dist-test-slave-rh5v krb5kdc[26832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:48.520418 26949 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:13:48.521632 26949 webserver.cc:460] Webserver started at http://127.24.141.1:44007/ using document root <none> and password file <none>
I20260620 12:13:48.522334 26949 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:48.522413 26949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:48.522629 26949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:48.525295 26949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "07d1a158beab4738bc511c65580cff0d"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "7394a8dddda5a29247a5868f6e3d209d"
server_key_iv: "05a94533a4617d0264805f41425c46eb"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.526010 26949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "07d1a158beab4738bc511c65580cff0d"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "7394a8dddda5a29247a5868f6e3d209d"
server_key_iv: "05a94533a4617d0264805f41425c46eb"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.530977 26949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 12:13:48.535331 26964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:48.537307 26949 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:13:48.537509 26949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "07d1a158beab4738bc511c65580cff0d"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "7394a8dddda5a29247a5868f6e3d209d"
server_key_iv: "05a94533a4617d0264805f41425c46eb"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.537649 26949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:48.577255 26949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:48.580960 26949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:48.581220 26949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:48.583043 26949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:48.583106 26949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:48.583168 26949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:48.583195 26949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:48.595705 26949 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:45697
I20260620 12:13:48.595837 27079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:45697 every 8 connection(s)
Jun 20 12:13:48 dist-test-slave-rh5v krb5kdc[26832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:48.596959 26949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:13:48.601420 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 26949
I20260620 12:13:48.601572 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:13:48.601948 25140 external_mini_cluster.cc:1428] Setting key 59be82f7f78f88b86d8faca544170ab7
I20260620 12:13:48.603060 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.585021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38009 (local address 127.24.141.1:48727)
0620 12:13:48.585649 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:48.585711 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:48.587108 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:48.587869 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:48.587883 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:48.588496 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:48.589534 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.589570 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.590907 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.590913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.592563 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.592585 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.593548 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.593555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.593842 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.594692 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:48.594719 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:48.596904 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:48.599270 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.599278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.599320 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.599696 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.600004 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.600007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.600009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.600171 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.600663 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:48.600672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:48.601009 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 12:13:48.601285 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 12:13:48.601762 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":158,"threads_started":1}
I20260620 12:13:48.604023 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.584471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:48727 (local address 127.24.141.62:38009)
0620 12:13:48.584678 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:48.584684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:48.585835 (+ 1151us) server_negotiation.cc:407] Connection header received
0620 12:13:48.587357 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:48.587361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:48.587432 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:48.587528 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:48.589820 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.590666 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.592812 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.593261 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.593427 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.597290 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:48.597296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:48.597300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:48.597341 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:48.599129 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.599818 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.599823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.599825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.599914 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.600272 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.600275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.600278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.600561 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:48.600655 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:48.603610 (+ 2955us) server_negotiation.cc:299] Negotiation successful
0620 12:13:48.603853 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 12:13:48.605373 27080 heartbeater.cc:347] Connected to a master server at 127.24.141.62:38009
I20260620 12:13:48.605558 27080 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:48.606258 27080 heartbeater.cc:511] Master 127.24.141.62:38009 requested a full tablet report, sending...
I20260620 12:13:48.608923 26876 ts_manager.cc:194] Registered new tserver with Master: 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697)
I20260620 12:13:48.610740 26876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:48727
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:13:48.618314 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.608781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:37537 (local address 127.24.141.62:38009)
0620 12:13:48.608957 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:48.608963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:48.609528 (+ 565us) server_negotiation.cc:407] Connection header received
0620 12:13:48.609865 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:48.609869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:48.609943 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:48.610060 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:48.610796 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.611662 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.612711 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.612961 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.613110 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.614656 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:48.614661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:48.614664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:48.614694 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:48.616430 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.616978 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.616983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.616985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.617108 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.617377 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.617381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.617383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.617595 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:48.617702 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:48.617998 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 12:13:48.618122 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:13:48.618716 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.608636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38009 (local address 127.24.141.1:37537)
0620 12:13:48.609377 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:48.609397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:48.609578 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:48.610136 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:48.610140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:48.610426 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:48.610624 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.610634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.611835 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.611839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.612523 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.612534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.613136 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.613140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.613246 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.613876 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:48.613894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:48.614431 (+ 537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:48.616574 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.616578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.616580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.616867 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.617197 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.617200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.617203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.617290 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.617681 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:48.617685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:48.617833 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 12:13:48.617925 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 12:13:48.618140 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":1,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:13:48.636204 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38009
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:48.772192 27089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:48.772535 27089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:48.772590 27089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:48.776999 27089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:48.777120 27089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:48.777173 27089 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:13:48.777189 27089 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:13:48.777233 27089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:13:48.783403 27089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:38009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:48.784868 27089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:48.786250 27089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:48.797699 27094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:48.797689 27095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:48.797689 27097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:48.798167 27089 server_base.cc:1034] running on GCE node
I20260620 12:13:48.798617 27089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:48.799396 27089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:48.800570 27089 hybrid_clock.cc:648] HybridClock initialized: now 1781957628800541 us; error 50 us; skew 500 ppm
Jun 20 12:13:48 dist-test-slave-rh5v krb5kdc[26832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:48.803721 27089 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:13:48.804857 27089 webserver.cc:460] Webserver started at http://127.24.141.2:34215/ using document root <none> and password file <none>
I20260620 12:13:48.805557 27089 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:48.805626 27089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:48.805860 27089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:48.808482 27089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "5618041561a1491d8c4c2e67e77cf55f"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "4c2fa731099a83686cf36aba75d20d53"
server_key_iv: "763646df02002f92335b95cd9b1ff979"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.809170 27089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "5618041561a1491d8c4c2e67e77cf55f"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "4c2fa731099a83686cf36aba75d20d53"
server_key_iv: "763646df02002f92335b95cd9b1ff979"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.814347 27089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 12:13:48.817835 27104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:48.819648 27089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:13:48.819794 27089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "5618041561a1491d8c4c2e67e77cf55f"
format_stamp: "Formatted at 2026-06-20 12:13:48 on dist-test-slave-rh5v"
server_key: "4c2fa731099a83686cf36aba75d20d53"
server_key_iv: "763646df02002f92335b95cd9b1ff979"
server_key_version: "encryptionkey@0"
I20260620 12:13:48.819948 27089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:48.836742 27089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:48.840420 27089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:48.840608 27089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:48.842571 27089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:48.842633 27089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:48.842694 27089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:48.842722 27089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:13:48 dist-test-slave-rh5v krb5kdc[26832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:48.854833 27089 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:36931
I20260620 12:13:48.855125 27219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:36931 every 8 connection(s)
I20260620 12:13:48.856101 27089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:13:48.859293 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.843896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:48965 (local address 127.24.141.62:38009)
0620 12:13:48.844079 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:48.844084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:48.845419 (+ 1335us) server_negotiation.cc:407] Connection header received
0620 12:13:48.846988 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:48.846991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:48.847056 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:48.847161 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:48.849230 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.849888 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.851525 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.851761 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.851852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.855337 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:48.855341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:48.855344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:48.855380 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:48.856888 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.857660 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.857665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.857667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.857793 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.858135 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.858137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.858139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.858376 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:48.858457 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:48.859005 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 12:13:48.859144 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:13:48.860617 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.844429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38009 (local address 127.24.141.2:48965)
0620 12:13:48.845178 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:48.845251 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:48.846667 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:48.847444 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:48.847456 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:48.848070 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:48.848986 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.849013 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.850080 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.850085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.851363 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.851382 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.851919 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.851923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.852155 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.852928 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:48.852956 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:48.855053 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:48.857092 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.857099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.857121 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.857496 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.857895 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.857897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.857899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.858046 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.858460 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:48.858466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:48.858759 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 12:13:48.858934 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 12:13:48.859379 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":178,"threads_started":1}
I20260620 12:13:48.862226 27220 heartbeater.cc:347] Connected to a master server at 127.24.141.62:38009
I20260620 12:13:48.862403 27220 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:48.863013 27220 heartbeater.cc:511] Master 127.24.141.62:38009 requested a full tablet report, sending...
I20260620 12:13:48.864284 26876 ts_manager.cc:194] Registered new tserver with Master: 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931)
I20260620 12:13:48.865105 26876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:48965
I20260620 12:13:48.865303 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 27089
I20260620 12:13:48.865425 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:13:48.865746 25140 external_mini_cluster.cc:1428] Setting key 66058d1b23b0a94246d940905ff82779
I20260620 12:13:48.875012 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.865114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:40549 (local address 127.24.141.62:38009)
0620 12:13:48.865259 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:48.865269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:48.865696 (+ 427us) server_negotiation.cc:407] Connection header received
0620 12:13:48.865822 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:48.865825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:48.865881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:48.865956 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:48.868442 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.869003 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.869905 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.870131 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.870223 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.871495 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:48.871498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:48.871501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:48.871524 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:48.873264 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.873721 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.873725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.873727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.873805 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:48.874063 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:48.874067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:48.874069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:48.874332 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:48.874433 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:48.874685 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 12:13:48.874809 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:13:48.875031 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:48.864953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38009 (local address 127.24.141.2:40549)
0620 12:13:48.865607 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:48.865622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:48.865731 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:48.867940 (+ 2209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:48.867943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:48.868171 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:48.868298 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.868305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.869161 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.869166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.869785 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:48.869794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:48.870249 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:48.870253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:48.870349 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:48.870960 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:48.870972 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:48.871356 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:48.873391 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.873395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.873397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.873624 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.873894 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:48.873897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:48.873900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:48.873980 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:48.874425 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:48.874428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:48.874532 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 12:13:48.874620 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:13:48.874808 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":174,"threads_started":1}
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:13:48.894212 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:38009
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:49.023383 27229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:49.023700 27229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:49.023751 27229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:49.028209 27229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:49.028303 27229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:49.028345 27229 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:13:49.028359 27229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:13:49.028393 27229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:13:49.034610 27229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:38009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:49.035794 27229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:49.036906 27229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:49.047961 27237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:49.048019 27235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:49.048019 27234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:49.048391 27229 server_base.cc:1034] running on GCE node
I20260620 12:13:49.049015 27229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:49.049834 27229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:49.051021 27229 hybrid_clock.cc:648] HybridClock initialized: now 1781957629050989 us; error 47 us; skew 500 ppm
Jun 20 12:13:49 dist-test-slave-rh5v krb5kdc[26832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957629, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:49.054481 27229 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:13:49.055609 27229 webserver.cc:460] Webserver started at http://127.24.141.3:37403/ using document root <none> and password file <none>
I20260620 12:13:49.056282 27229 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:49.056360 27229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:49.056571 27229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:49.058835 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "557c40d87c194c8889eec2e653935455"
format_stamp: "Formatted at 2026-06-20 12:13:49 on dist-test-slave-rh5v"
server_key: "3f8dd8b804e9dc0f0e1588750ddcaefb"
server_key_iv: "5ddb68744bbaf5e51135b03c0aafdd59"
server_key_version: "encryptionkey@0"
I20260620 12:13:49.059458 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "557c40d87c194c8889eec2e653935455"
format_stamp: "Formatted at 2026-06-20 12:13:49 on dist-test-slave-rh5v"
server_key: "3f8dd8b804e9dc0f0e1588750ddcaefb"
server_key_iv: "5ddb68744bbaf5e51135b03c0aafdd59"
server_key_version: "encryptionkey@0"
I20260620 12:13:49.063923 27229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 12:13:49.066954 27244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:49.068979 27229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:49.069216 27229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "557c40d87c194c8889eec2e653935455"
format_stamp: "Formatted at 2026-06-20 12:13:49 on dist-test-slave-rh5v"
server_key: "3f8dd8b804e9dc0f0e1588750ddcaefb"
server_key_iv: "5ddb68744bbaf5e51135b03c0aafdd59"
server_key_version: "encryptionkey@0"
I20260620 12:13:49.069387 27229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:49.091449 27229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:49.095510 27229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:49.095741 27229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:49.097685 27229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:49.097744 27229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:49.097806 27229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:49.097833 27229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:13:49 dist-test-slave-rh5v krb5kdc[26832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957629, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:49.110052 27229 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:34153
I20260620 12:13:49.110140 27359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:34153 every 8 connection(s)
I20260620 12:13:49.111327 27229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:13:49.112605 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 27229
I20260620 12:13:49.112740 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:13:49.113109 25140 external_mini_cluster.cc:1428] Setting key 15a7f2922ec3f625243fa25f27f684d1
I20260620 12:13:49.114593 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.098869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:60915 (local address 127.24.141.62:38009)
0620 12:13:49.099003 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.099008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.100621 (+ 1613us) server_negotiation.cc:407] Connection header received
0620 12:13:49.101948 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.101952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.102018 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.102099 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:49.103653 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.104365 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.105815 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.106100 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.106276 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.110001 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:49.110006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:49.110009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:49.110044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:49.111730 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.112486 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.112492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.112495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.112642 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.113167 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.113170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.113172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.113495 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:49.113600 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.114221 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.114449 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:13:49.115772 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.099669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38009 (local address 127.24.141.3:60915)
0620 12:13:49.100436 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.100498 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.101697 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.102325 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.102333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.102845 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:49.103451 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.103475 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.104513 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.104517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.105663 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.105675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.106465 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.106469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.106684 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.107406 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:49.107432 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:49.109747 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:49.111902 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:49.111910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:49.111932 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:49.112303 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:49.112825 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:49.112830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:49.112832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:49.113017 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:49.113624 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:49.113632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:49.113966 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.114174 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.114558 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":170,"threads_started":1}
I20260620 12:13:49.117213 27360 heartbeater.cc:347] Connected to a master server at 127.24.141.62:38009
I20260620 12:13:49.117414 27360 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:49.118150 27360 heartbeater.cc:511] Master 127.24.141.62:38009 requested a full tablet report, sending...
I20260620 12:13:49.119529 26877 ts_manager.cc:194] Registered new tserver with Master: 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153)
I20260620 12:13:49.120308 26877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:60915
I20260620 12:13:49.127990 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:13:49.128146 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.119938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:37625 (local address 127.24.141.62:38009)
0620 12:13:49.120073 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.120078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.120337 (+ 259us) server_negotiation.cc:407] Connection header received
0620 12:13:49.120541 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.120544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.120601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.120684 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:49.121270 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.122054 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.123037 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.123310 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.123401 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.124623 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:49.124626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:49.124628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:49.124652 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:49.126187 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.126695 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.126699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.126701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.126797 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.127163 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.127166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.127168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.127391 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:49.127476 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.127856 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.127981 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:13:49.128166 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.119829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38009 (local address 127.24.141.3:37625)
0620 12:13:49.120219 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.120237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.120381 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.120726 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.120729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.120967 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:49.121144 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.121153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.122204 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.122207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.122906 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.122919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.123418 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.123420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.123494 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.124107 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:49.124121 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:49.124484 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:49.126326 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:49.126329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:49.126330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:49.126593 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:49.126927 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:49.126931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:49.126933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:49.127054 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:49.127505 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:49.127508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:49.127646 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.127774 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.127956 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":165,"threads_started":1}
Jun 20 12:13:49 dist-test-slave-rh5v krb5kdc[26832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:13:49 dist-test-slave-rh5v krb5kdc[26832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:49.157361 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.148466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.24.141.62:38009)
0620 12:13:49.148656 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.148660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.148675 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:13:49.148726 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.148729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.148776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.148864 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:49.149831 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.150413 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.151343 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.151563 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.151654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.154409 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:49.154414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:49.154416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:49.154445 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:49.155753 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.156152 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.156157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.156159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.156233 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.156475 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.156478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.156480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.156669 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:49.156792 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.157015 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.157171 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 12:13:49.160959 26877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:13:49.164793 26877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:49.183884 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.177506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.24.141.2:36931)
0620 12:13:49.178326 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.178340 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.178735 (+ 395us) server_negotiation.cc:407] Connection header received
0620 12:13:49.178834 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.178840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.179102 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.179272 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.179908 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.180718 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.182244 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.182937 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.183053 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.183248 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.183552 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.183677 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":181,"threads_started":1}
I20260620 12:13:49.183902 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.177218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.0.0.1:53302)
0620 12:13:49.178611 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.178623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.178695 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.179321 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.179328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.179372 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.179773 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.179779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.180834 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.180838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.182118 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.182128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.183101 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.183105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.183280 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.183342 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.183501 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.183670 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":199,"spinlock_wait_cycles":4736,"thread_start_us":101,"threads_started":1}
I20260620 12:13:49.184345 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.177599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45697 (local address 127.0.0.1:37216)
0620 12:13:49.178218 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.178281 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.178438 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.179321 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.179328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.179375 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.179757 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.179766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.181497 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.181501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.182731 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.182740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.183895 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.183900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.184028 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.184062 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.184122 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.184206 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":121,"spinlock_wait_cycles":7424,"thread_start_us":120,"threads_started":1}
I20260620 12:13:49.184671 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.177634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34153 (local address 127.0.0.1:45558)
0620 12:13:49.178508 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.178521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.178599 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.179485 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.179488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.179501 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.179728 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.179738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.181287 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.181291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.182646 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.182662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.183911 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.183915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.184016 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.184333 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.184424 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.184528 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":12,"thread_start_us":128,"threads_started":1}
I20260620 12:13:49.185240 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.178217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.24.141.3:34153)
0620 12:13:49.178963 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.178976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.178998 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:13:49.179071 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.179076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.179250 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.179383 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.179896 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.181112 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.182833 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.183582 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.183688 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.183837 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.184907 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.185031 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":122,"threads_started":1}
I20260620 12:13:49.185653 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.177254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.24.141.1:45697)
0620 12:13:49.178783 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.178793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.178827 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:13:49.178923 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.178928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.179145 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.179579 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.179915 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.181357 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.182895 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.183735 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.185117 (+ 1382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.185333 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.185397 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.185516 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1391,"thread_start_us":129,"threads_started":1}
I20260620 12:13:49.188334 27154 tablet_service.cc:1467] Processing CreateTablet for tablet c15462a262ae4a65ab167576ac908831 (DEFAULT_TABLE table=test-table [id=3079ec1609e442a7b542dc8b02a63215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:49.188601 27014 tablet_service.cc:1467] Processing CreateTablet for tablet c15462a262ae4a65ab167576ac908831 (DEFAULT_TABLE table=test-table [id=3079ec1609e442a7b542dc8b02a63215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:49.188614 27291 tablet_service.cc:1467] Processing CreateTablet for tablet c15462a262ae4a65ab167576ac908831 (DEFAULT_TABLE table=test-table [id=3079ec1609e442a7b542dc8b02a63215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:49.189577 27154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15462a262ae4a65ab167576ac908831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.189863 27014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15462a262ae4a65ab167576ac908831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.189879 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15462a262ae4a65ab167576ac908831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.198475 27383 tablet_bootstrap.cc:492] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Bootstrap starting.
I20260620 12:13:49.199883 27385 tablet_bootstrap.cc:492] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Bootstrap starting.
I20260620 12:13:49.201294 27384 tablet_bootstrap.cc:492] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Bootstrap starting.
I20260620 12:13:49.201406 27383 tablet_bootstrap.cc:654] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.202553 27385 tablet_bootstrap.cc:654] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.202737 27383 log.cc:824] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:49.203570 27385 log.cc:824] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:49.204085 27384 tablet_bootstrap.cc:654] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.205174 27384 log.cc:824] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:49.205749 27383 tablet_bootstrap.cc:492] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: No bootstrap required, opened a new log
I20260620 12:13:49.205806 27385 tablet_bootstrap.cc:492] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: No bootstrap required, opened a new log
I20260620 12:13:49.205989 27383 ts_tablet_manager.cc:1397] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:13:49.206027 27385 ts_tablet_manager.cc:1397] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:13:49.207839 27384 tablet_bootstrap.cc:492] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: No bootstrap required, opened a new log
I20260620 12:13:49.208124 27384 ts_tablet_manager.cc:1397] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 12:13:49.210021 27385 raft_consensus.cc:348] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.210412 27385 raft_consensus.cc:374] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.210474 27385 raft_consensus.cc:729] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557c40d87c194c8889eec2e653935455, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.210729 27383 raft_consensus.cc:348] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.210976 27385 consensus_queue.cc:260] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.211074 27383 raft_consensus.cc:374] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.211130 27383 raft_consensus.cc:729] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5618041561a1491d8c4c2e67e77cf55f, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.211719 27383 consensus_queue.cc:260] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.212285 27385 ts_tablet_manager.cc:1428] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:13:49.212668 27360 heartbeater.cc:503] Master 127.24.141.62:38009 was elected leader, sending a full tablet report...
I20260620 12:13:49.212675 27220 heartbeater.cc:503] Master 127.24.141.62:38009 was elected leader, sending a full tablet report...
I20260620 12:13:49.212625 27384 raft_consensus.cc:348] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.212908 27384 raft_consensus.cc:374] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.212970 27384 raft_consensus.cc:729] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d1a158beab4738bc511c65580cff0d, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.213160 27383 ts_tablet_manager.cc:1428] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:13:49.213558 27384 consensus_queue.cc:260] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.214520 27080 heartbeater.cc:503] Master 127.24.141.62:38009 was elected leader, sending a full tablet report...
I20260620 12:13:49.215867 27384 ts_tablet_manager.cc:1428] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 12:13:49.239796 27391 raft_consensus.cc:482] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:49.240130 27391 raft_consensus.cc:504] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.241784 27391 leader_election.cc:288] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931), 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153)
I20260620 12:13:49.247927 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.242256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:44109 (local address 127.24.141.2:36931)
0620 12:13:49.242423 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.242428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.242724 (+ 296us) server_negotiation.cc:407] Connection header received
0620 12:13:49.243029 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.243034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.243104 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.243182 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.243504 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.244392 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.246404 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.247114 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.247265 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.247326 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.247661 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.247768 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:13:49.247927 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.242543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:57839 (local address 127.24.141.3:34153)
0620 12:13:49.242691 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.242697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.243000 (+ 303us) server_negotiation.cc:407] Connection header received
0620 12:13:49.243235 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.243239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.243298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.243373 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.243709 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.244469 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.246404 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.247113 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.247265 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.247326 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.247661 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.247768 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:13:49.247967 27393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.242336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34153 (local address 127.24.141.1:57839)
0620 12:13:49.242884 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.242901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.243082 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.243408 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.243411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.243431 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.243559 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.243565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.244588 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.244591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.246244 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.246262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.247271 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.247276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.247458 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.247494 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.247580 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.247723 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":145,"thread_start_us":156,"threads_started":1}
I20260620 12:13:49.247973 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.242131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.24.141.1:44109)
0620 12:13:49.242617 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.242636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.242819 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.243212 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.243214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.243228 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.243347 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.243352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.244556 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.244560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.246244 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.246258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.247271 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.247276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.247423 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.247499 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.247580 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.247723 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":190,"thread_start_us":171,"threads_started":1}
I20260620 12:13:49.248837 27314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15462a262ae4a65ab167576ac908831" candidate_uuid: "07d1a158beab4738bc511c65580cff0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557c40d87c194c8889eec2e653935455" is_pre_election: true
I20260620 12:13:49.249315 27314 raft_consensus.cc:2454] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d1a158beab4738bc511c65580cff0d in term 0.
I20260620 12:13:49.249637 27174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15462a262ae4a65ab167576ac908831" candidate_uuid: "07d1a158beab4738bc511c65580cff0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5618041561a1491d8c4c2e67e77cf55f" is_pre_election: true
I20260620 12:13:49.249975 26966 leader_election.cc:302] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 557c40d87c194c8889eec2e653935455; no voters:
I20260620 12:13:49.250037 27174 raft_consensus.cc:2454] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d1a158beab4738bc511c65580cff0d in term 0.
I20260620 12:13:49.250293 27391 raft_consensus.cc:2781] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:49.250406 27391 raft_consensus.cc:482] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:49.250454 27391 raft_consensus.cc:3037] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.251814 27391 raft_consensus.cc:504] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.252270 27391 leader_election.cc:288] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [CANDIDATE]: Term 1 election: Requested vote from peers 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931), 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153)
I20260620 12:13:49.252784 27174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15462a262ae4a65ab167576ac908831" candidate_uuid: "07d1a158beab4738bc511c65580cff0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5618041561a1491d8c4c2e67e77cf55f"
I20260620 12:13:49.252784 27314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15462a262ae4a65ab167576ac908831" candidate_uuid: "07d1a158beab4738bc511c65580cff0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557c40d87c194c8889eec2e653935455"
I20260620 12:13:49.252964 27174 raft_consensus.cc:3037] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.252964 27314 raft_consensus.cc:3037] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.254459 27314 raft_consensus.cc:2454] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d1a158beab4738bc511c65580cff0d in term 1.
I20260620 12:13:49.254459 27174 raft_consensus.cc:2454] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d1a158beab4738bc511c65580cff0d in term 1.
I20260620 12:13:49.254868 26966 leader_election.cc:302] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 557c40d87c194c8889eec2e653935455; no voters:
I20260620 12:13:49.255103 27391 raft_consensus.cc:2781] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:49.255378 27391 raft_consensus.cc:686] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 1 LEADER]: Becoming Leader. State: Replica: 07d1a158beab4738bc511c65580cff0d, State: Running, Role: LEADER
I20260620 12:13:49.255798 27391 consensus_queue.cc:237] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.259727 26877 catalog_manager.cc:5495] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 07d1a158beab4738bc511c65580cff0d (127.24.141.1). New cstate: current_term: 1 leader_uuid: "07d1a158beab4738bc511c65580cff0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } health_report { overall_health: UNKNOWN } } }
Jun 20 12:13:49 dist-test-slave-rh5v krb5kdc[26832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:49.309211 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.299501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38009 (local address 127.0.0.1:45614)
0620 12:13:49.299864 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.299879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.300043 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.300287 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.300290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.300797 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:49.300912 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.300919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.301740 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.301744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.302317 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.302325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.302733 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.302736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.302824 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.303497 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:49.303508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:49.305259 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:49.307180 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:49.307189 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:49.307197 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:49.307480 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:49.307889 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:49.307892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:49.307893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:49.307985 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:49.308498 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:49.308505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:49.308639 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.308758 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.308945 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260620 12:13:49.309325 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.299573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.24.141.62:38009)
0620 12:13:49.299711 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.299716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.299975 (+ 259us) server_negotiation.cc:407] Connection header received
0620 12:13:49.300134 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.300137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.300179 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.300247 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:49.301027 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.301638 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.302428 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.302638 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.302721 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.305505 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:49.305510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:49.305512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:49.305553 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:49.307038 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.307625 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.307629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.307630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.307766 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:49.308118 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:49.308122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:49.308123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:49.308359 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:49.308464 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.308883 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.309098 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:13:49.314487 26877 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:13:49.315423 26877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:49.321412 27154 tablet_service.cc:1467] Processing CreateTablet for tablet 90d8f611dfc84ce7b81af6f23060d5ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c4274b23d0743f7ae5c70c68352c403]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:13:49.321691 27291 tablet_service.cc:1467] Processing CreateTablet for tablet 90d8f611dfc84ce7b81af6f23060d5ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c4274b23d0743f7ae5c70c68352c403]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:13:49.321502 27014 tablet_service.cc:1467] Processing CreateTablet for tablet 90d8f611dfc84ce7b81af6f23060d5ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c4274b23d0743f7ae5c70c68352c403]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:13:49.322206 27154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d8f611dfc84ce7b81af6f23060d5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.322453 27014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d8f611dfc84ce7b81af6f23060d5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.322453 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d8f611dfc84ce7b81af6f23060d5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.325172 27383 tablet_bootstrap.cc:492] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: Bootstrap starting.
I20260620 12:13:49.326145 27385 tablet_bootstrap.cc:492] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: Bootstrap starting.
I20260620 12:13:49.326244 27384 tablet_bootstrap.cc:492] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: Bootstrap starting.
I20260620 12:13:49.326670 27383 tablet_bootstrap.cc:654] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.327195 27385 tablet_bootstrap.cc:654] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.327344 27384 tablet_bootstrap.cc:654] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.328270 27383 tablet_bootstrap.cc:492] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: No bootstrap required, opened a new log
I20260620 12:13:49.328372 27383 ts_tablet_manager.cc:1397] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:49.328709 27385 tablet_bootstrap.cc:492] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: No bootstrap required, opened a new log
I20260620 12:13:49.328814 27385 ts_tablet_manager.cc:1397] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:49.328878 27383 raft_consensus.cc:348] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.329017 27383 raft_consensus.cc:374] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.329090 27383 raft_consensus.cc:729] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5618041561a1491d8c4c2e67e77cf55f, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.329420 27384 tablet_bootstrap.cc:492] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: No bootstrap required, opened a new log
I20260620 12:13:49.329356 27385 raft_consensus.cc:348] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.329373 27383 consensus_queue.cc:260] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.329471 27385 raft_consensus.cc:374] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.329502 27385 raft_consensus.cc:729] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557c40d87c194c8889eec2e653935455, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.329520 27384 ts_tablet_manager.cc:1397] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:49.329684 27383 ts_tablet_manager.cc:1428] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:13:49.329668 27385 consensus_queue.cc:260] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.329947 27389 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.329993 27384 raft_consensus.cc:348] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.330107 27389 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:49.330113 27384 raft_consensus.cc:374] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.330160 27384 raft_consensus.cc:729] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d1a158beab4738bc511c65580cff0d, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.330282 27384 consensus_queue.cc:260] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.330349 27385 ts_tablet_manager.cc:1428] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:13:49.330535 27384 ts_tablet_manager.cc:1428] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:13:49.330814 27390 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.330976 27390 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:49.331799 27391 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.331972 27391 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 12:13:49.348770 27081 tablet.cc:2369] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:13:49.357483 27221 tablet.cc:2369] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:13:49.362610 27361 tablet.cc:2369] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:13:49.500936 27390 raft_consensus.cc:482] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:49.501191 27390 raft_consensus.cc:504] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.502665 27390 leader_election.cc:288] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153)
I20260620 12:13:49.507326 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.502946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45697 (local address 127.24.141.2:45597)
0620 12:13:49.503269 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.503286 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.503413 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.503784 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.503787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.503806 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.503988 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.503996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.504917 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.504921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.505931 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.505940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.506795 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.506799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.506923 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.506962 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.507068 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.507169 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":85,"threads_started":1}
I20260620 12:13:49.507401 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.503407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:45597 (local address 127.24.141.1:45697)
0620 12:13:49.503521 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.503525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.503542 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:13:49.503608 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.503612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.503672 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.503760 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.504141 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.504793 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.506076 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.506636 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.506761 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.506816 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.507126 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.507224 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 12:13:49.507787 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.503132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:34531 (local address 127.24.141.3:34153)
0620 12:13:49.503272 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.503277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.503727 (+ 450us) server_negotiation.cc:407] Connection header received
0620 12:13:49.503876 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.503879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.503939 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.504042 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.504439 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.505004 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.506464 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.507002 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.507116 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.507169 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.507524 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.507682 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:13:49.508249 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.503093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34153 (local address 127.24.141.2:34531)
0620 12:13:49.503604 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.503619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.503767 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.504116 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.504121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.504147 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.504297 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.504304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.505202 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.505206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.506242 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.506259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.507146 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.507151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.507283 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.507323 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.507964 (+ 641us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.508084 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":105,"threads_started":1}
I20260620 12:13:49.508376 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d8f611dfc84ce7b81af6f23060d5ec" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d" is_pre_election: true
I20260620 12:13:49.508726 27034 raft_consensus.cc:2454] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 0.
I20260620 12:13:49.508983 27314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d8f611dfc84ce7b81af6f23060d5ec" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557c40d87c194c8889eec2e653935455" is_pre_election: true
I20260620 12:13:49.509196 27314 raft_consensus.cc:2454] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 0.
I20260620 12:13:49.509349 27107 leader_election.cc:302] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 5618041561a1491d8c4c2e67e77cf55f; no voters:
I20260620 12:13:49.509769 27390 raft_consensus.cc:2781] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:49.509861 27390 raft_consensus.cc:482] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:49.509889 27390 raft_consensus.cc:3037] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.510974 27390 raft_consensus.cc:504] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.511451 27390 leader_election.cc:288] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 election: Requested vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153)
I20260620 12:13:49.511915 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d8f611dfc84ce7b81af6f23060d5ec" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d"
I20260620 12:13:49.511951 27314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d8f611dfc84ce7b81af6f23060d5ec" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557c40d87c194c8889eec2e653935455"
I20260620 12:13:49.512049 27034 raft_consensus.cc:3037] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.512097 27314 raft_consensus.cc:3037] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.513278 27314 raft_consensus.cc:2454] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 1.
I20260620 12:13:49.513365 27034 raft_consensus.cc:2454] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 1.
I20260620 12:13:49.513681 27106 leader_election.cc:302] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 557c40d87c194c8889eec2e653935455, 5618041561a1491d8c4c2e67e77cf55f; no voters:
I20260620 12:13:49.513921 27390 raft_consensus.cc:2781] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:49.514125 27390 raft_consensus.cc:686] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [term 1 LEADER]: Becoming Leader. State: Replica: 5618041561a1491d8c4c2e67e77cf55f, State: Running, Role: LEADER
I20260620 12:13:49.514568 27390 consensus_queue.cc:237] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.517588 27407 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: TxnStatusTablet state changed. Reason: New leader 5618041561a1491d8c4c2e67e77cf55f. Latest consensus state: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.517771 27407 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:13:49.518132 27409 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:13:49.518437 27409 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:13:49.519091 26875 catalog_manager.cc:5495] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2). New cstate: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } health_report { overall_health: UNKNOWN } } }
I20260620 12:13:49.535445 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.530165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.24.141.2:36931)
0620 12:13:49.530448 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.530454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.530482 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:13:49.530600 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.530604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.530692 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.530850 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:49.531141 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.531926 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.532925 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.533367 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.533503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.534205 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.534893 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.535002 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.535170 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.535258 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260620 12:13:49.535492 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.530073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.0.0.1:53308)
0620 12:13:49.530284 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.530301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.530428 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.530874 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.530877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.530884 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:49.531018 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.531023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.532084 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.532087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.532792 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.532801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.533656 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.533662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.533785 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.533869 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.535015 (+ 1146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.535046 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.535156 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.535297 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 12:13:49.542367 27034 raft_consensus.cc:1261] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Refusing update from remote peer 5618041561a1491d8c4c2e67e77cf55f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:49.542537 27314 raft_consensus.cc:1261] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Refusing update from remote peer 5618041561a1491d8c4c2e67e77cf55f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:49.543286 27407 consensus_queue.cc:1035] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:49.543581 27390 consensus_queue.cc:1035] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:49.547719 27391 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: TxnStatusTablet state changed. Reason: New leader 5618041561a1491d8c4c2e67e77cf55f. Latest consensus state: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.547896 27391 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:49.548955 27389 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: TxnStatusTablet state changed. Reason: New leader 5618041561a1491d8c4c2e67e77cf55f. Latest consensus state: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.549106 27389 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:49.553104 27390 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.553260 27390 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:13:49.553401 27390 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.553484 27390 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 5618041561a1491d8c4c2e67e77cf55f: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:13:49.553566 27412 mvcc.cc:205] Tried to move back new op lower bound from 7298898450596446208 to 7298898450495070208. Current Snapshot: MvccSnapshot[applied={T|T < 7298898450596446208}]
I20260620 12:13:49.554241 27389 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.554370 27389 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 557c40d87c194c8889eec2e653935455: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:49.554661 27391 tablet_replica.cc:439] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } } }
I20260620 12:13:49.554853 27391 tablet_replica.cc:442] T 90d8f611dfc84ce7b81af6f23060d5ec P 07d1a158beab4738bc511c65580cff0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:13:49.555683 27414 mvcc.cc:205] Tried to move back new op lower bound from 7298898450596446208 to 7298898450495070208. Current Snapshot: MvccSnapshot[applied={T|T < 7298898450596446208}]
I20260620 12:13:49.555898 27413 mvcc.cc:205] Tried to move back new op lower bound from 7298898450596446208 to 7298898450495070208. Current Snapshot: MvccSnapshot[applied={T|T < 7298898450596446208}]
I20260620 12:13:49.568502 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.563831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.24.141.1:45697)
0620 12:13:49.564048 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.564053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.564075 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:13:49.564185 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.564189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.564266 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.564404 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:49.564631 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.565454 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.566558 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.566933 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.567051 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.567342 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.568034 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.568160 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.568268 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.568340 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 12:13:49.582419 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.578766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:49921 (local address 127.24.141.2:36931)
0620 12:13:49.578919 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.578925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.579111 (+ 186us) server_negotiation.cc:407] Connection header received
0620 12:13:49.579274 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.579277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.579326 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.579393 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:49.579689 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.580248 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.581173 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.581446 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.581540 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.581904 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.581996 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.582054 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.582151 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.582219 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:13:49.582434 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.578639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.24.141.1:49921)
0620 12:13:49.579017 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.579031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.579167 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.579446 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.579449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.579457 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:49.579596 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.579601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.580349 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.580360 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.581037 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.581046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.581554 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.581558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.581657 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.581722 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.582067 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.582074 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.582136 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.582235 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":145,"threads_started":1}
I20260620 12:13:49.592016 27314 raft_consensus.cc:1261] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Refusing update from remote peer 07d1a158beab4738bc511c65580cff0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:49.592442 27174 raft_consensus.cc:1261] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Refusing update from remote peer 07d1a158beab4738bc511c65580cff0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:49.592655 27391 consensus_queue.cc:1035] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:49.593010 27394 consensus_queue.cc:1035] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:49.632985 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.627351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45697 (local address 127.24.141.2:55529)
0620 12:13:49.627766 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.627788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.627961 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.628799 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.628804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.628817 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:49.628945 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.628952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.629988 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.629993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.631079 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.631092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.631692 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.631696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.631791 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.631865 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.632467 (+ 602us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.632475 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.632544 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.632667 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260620 12:13:49.633280 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.627558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:55529 (local address 127.24.141.1:45697)
0620 12:13:49.627698 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.627702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.627975 (+ 273us) server_negotiation.cc:407] Connection header received
0620 12:13:49.628375 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.628379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.628445 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.628534 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:49.629135 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.629826 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.631284 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.631534 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.631665 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.632116 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.632248 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.632407 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.632999 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.633118 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:13:49.669786 26875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:49.670032 26875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:49.673454 26875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Sending DeleteTablet for 3 replicas of tablet c15462a262ae4a65ab167576ac908831
I20260620 12:13:49.675022 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet c15462a262ae4a65ab167576ac908831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:49 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45558
I20260620 12:13:49.675756 27432 tablet_replica.cc:330] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: stopping tablet replica
I20260620 12:13:49.675943 27014 tablet_service.cc:1514] Processing DeleteTablet for tablet c15462a262ae4a65ab167576ac908831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:49 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:37216
I20260620 12:13:49.676388 27432 raft_consensus.cc:2229] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:49.676487 27154 tablet_service.cc:1514] Processing DeleteTablet for tablet c15462a262ae4a65ab167576ac908831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:49 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:53302
I20260620 12:13:49.676882 27432 raft_consensus.cc:2258] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:49.678208 26875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:13:49.678848 26875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:49.678884 27432 ts_tablet_manager.cc:1905] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:49.682852 27432 ts_tablet_manager.cc:1918] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:49.683143 27432 log.cc:1196] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/c15462a262ae4a65ab167576ac908831
I20260620 12:13:49.683549 27432 ts_tablet_manager.cc:1939] T c15462a262ae4a65ab167576ac908831 P 557c40d87c194c8889eec2e653935455: Deleting consensus metadata
I20260620 12:13:49.685832 26862 catalog_manager.cc:4841] TS 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153): tablet c15462a262ae4a65ab167576ac908831 (table test-table [id=3079ec1609e442a7b542dc8b02a63215]) successfully deleted
I20260620 12:13:49.686017 27154 tablet_service.cc:1467] Processing CreateTablet for tablet 55fcca1aa0d340babee0158f47d74eac (DEFAULT_TABLE table=test-table [id=99cddfae2b62457d8550ca10321131dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:49.686365 27291 tablet_service.cc:1467] Processing CreateTablet for tablet 55fcca1aa0d340babee0158f47d74eac (DEFAULT_TABLE table=test-table [id=99cddfae2b62457d8550ca10321131dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:49.686515 27154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fcca1aa0d340babee0158f47d74eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.686705 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fcca1aa0d340babee0158f47d74eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.689955 27383 tablet_bootstrap.cc:492] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: Bootstrap starting.
I20260620 12:13:49.689972 27014 tablet_service.cc:1467] Processing CreateTablet for tablet 55fcca1aa0d340babee0158f47d74eac (DEFAULT_TABLE table=test-table [id=99cddfae2b62457d8550ca10321131dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:49.690351 27014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fcca1aa0d340babee0158f47d74eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:49.691115 27383 tablet_bootstrap.cc:654] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.693295 27383 tablet_bootstrap.cc:492] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: No bootstrap required, opened a new log
I20260620 12:13:49.693428 27383 ts_tablet_manager.cc:1397] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 12:13:49.693792 27385 tablet_bootstrap.cc:492] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: Bootstrap starting.
I20260620 12:13:49.693933 27383 raft_consensus.cc:348] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.694070 27383 raft_consensus.cc:374] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.694101 27383 raft_consensus.cc:729] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5618041561a1491d8c4c2e67e77cf55f, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.694236 27383 consensus_queue.cc:260] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.694502 27383 ts_tablet_manager.cc:1428] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:13:49.694921 27385 tablet_bootstrap.cc:654] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.696549 27384 tablet_bootstrap.cc:492] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: Bootstrap starting.
I20260620 12:13:49.697731 27384 tablet_bootstrap.cc:654] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:49.699544 27384 tablet_bootstrap.cc:492] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: No bootstrap required, opened a new log
I20260620 12:13:49.699654 27384 ts_tablet_manager.cc:1397] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 12:13:49.700222 27384 raft_consensus.cc:348] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.700377 27384 raft_consensus.cc:374] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.700407 27384 raft_consensus.cc:729] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d1a158beab4738bc511c65580cff0d, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.700579 27384 consensus_queue.cc:260] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.700975 27384 ts_tablet_manager.cc:1428] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 12:13:49.704783 27385 tablet_bootstrap.cc:492] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: No bootstrap required, opened a new log
I20260620 12:13:49.704921 27385 ts_tablet_manager.cc:1397] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260620 12:13:49.705657 27385 raft_consensus.cc:348] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.705803 27385 raft_consensus.cc:374] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:49.705833 27385 raft_consensus.cc:729] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557c40d87c194c8889eec2e653935455, State: Initialized, Role: FOLLOWER
I20260620 12:13:49.705982 27385 consensus_queue.cc:260] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.706286 27385 ts_tablet_manager.cc:1428] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:13:49.709192 27433 tablet_replica.cc:330] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: stopping tablet replica
I20260620 12:13:49.709702 27433 raft_consensus.cc:2229] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:13:49.710203 27433 raft_consensus.cc:2258] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:49.710422 27434 tablet_replica.cc:330] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: stopping tablet replica
I20260620 12:13:49.711578 27434 raft_consensus.cc:2229] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:49.711799 27434 raft_consensus.cc:2258] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:49.712318 27433 ts_tablet_manager.cc:1905] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:49.714670 27434 ts_tablet_manager.cc:1905] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:49.716915 27433 ts_tablet_manager.cc:1918] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:49.717180 27433 log.cc:1196] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/c15462a262ae4a65ab167576ac908831
I20260620 12:13:49.717705 27433 ts_tablet_manager.cc:1939] T c15462a262ae4a65ab167576ac908831 P 07d1a158beab4738bc511c65580cff0d: Deleting consensus metadata
I20260620 12:13:49.719421 26863 catalog_manager.cc:4841] TS 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697): tablet c15462a262ae4a65ab167576ac908831 (table test-table [id=3079ec1609e442a7b542dc8b02a63215]) successfully deleted
I20260620 12:13:49.719959 27434 ts_tablet_manager.cc:1918] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:49.720082 27434 log.cc:1196] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/c15462a262ae4a65ab167576ac908831
I20260620 12:13:49.720413 27434 ts_tablet_manager.cc:1939] T c15462a262ae4a65ab167576ac908831 P 5618041561a1491d8c4c2e67e77cf55f: Deleting consensus metadata
I20260620 12:13:49.722545 26861 catalog_manager.cc:4841] TS 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931): tablet c15462a262ae4a65ab167576ac908831 (table test-table [id=3079ec1609e442a7b542dc8b02a63215]) successfully deleted
I20260620 12:13:49.767015 27389 raft_consensus.cc:482] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:49.767232 27389 raft_consensus.cc:504] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.768674 27389 leader_election.cc:288] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931)
I20260620 12:13:49.773923 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.769509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:58501 (local address 127.24.141.1:45697)
0620 12:13:49.769650 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.769655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.769670 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:13:49.769869 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.769872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.769928 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.770005 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.770345 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.771110 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.772675 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.773184 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.773300 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.773402 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.773647 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.773741 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:13:49.774138 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.769184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:42947 (local address 127.24.141.2:36931)
0620 12:13:49.769334 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.769339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.769627 (+ 288us) server_negotiation.cc:407] Connection header received
0620 12:13:49.769925 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.769929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.769993 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.770074 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:13:49.770364 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.770951 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.772667 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.773457 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.773621 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.773689 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.773876 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.774016 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:13:49.774430 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.769036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45697 (local address 127.24.141.3:58501)
0620 12:13:49.769532 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.769550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.769722 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.770018 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.770021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.770036 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.770182 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.770190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.771260 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.771264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.772486 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.772506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.773300 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.773304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.773485 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.773516 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.774163 (+ 647us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.774258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260620 12:13:49.774902 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.769012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.24.141.3:42947)
0620 12:13:49.769470 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.769489 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.769753 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.770160 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.770163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.770176 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:13:49.770271 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.770277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.771061 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.771065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.772511 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.772517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.773635 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.773639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.773731 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.773753 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.774388 (+ 635us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.774462 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":46,"thread_start_us":120,"threads_started":1}
I20260620 12:13:49.775153 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcca1aa0d340babee0158f47d74eac" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d" is_pre_election: true
I20260620 12:13:49.775346 27034 raft_consensus.cc:2454] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 0.
I20260620 12:13:49.775359 27174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcca1aa0d340babee0158f47d74eac" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5618041561a1491d8c4c2e67e77cf55f" is_pre_election: true
I20260620 12:13:49.775547 27174 raft_consensus.cc:2454] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 0.
I20260620 12:13:49.775799 27247 leader_election.cc:302] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 557c40d87c194c8889eec2e653935455; no voters:
I20260620 12:13:49.776176 27389 raft_consensus.cc:2781] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:49.776238 27389 raft_consensus.cc:482] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:49.776284 27389 raft_consensus.cc:3037] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.777534 27389 raft_consensus.cc:504] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.777953 27389 leader_election.cc:288] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 election: Requested vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931)
I20260620 12:13:49.778431 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcca1aa0d340babee0158f47d74eac" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d"
I20260620 12:13:49.778612 27034 raft_consensus.cc:3037] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.779950 27034 raft_consensus.cc:2454] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 1.
I20260620 12:13:49.780337 27247 leader_election.cc:302] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 557c40d87c194c8889eec2e653935455; no voters:
I20260620 12:13:49.780333 27174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcca1aa0d340babee0158f47d74eac" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5618041561a1491d8c4c2e67e77cf55f"
I20260620 12:13:49.780550 27389 raft_consensus.cc:2781] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:49.780550 27174 raft_consensus.cc:3037] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:49.780745 27389 raft_consensus.cc:686] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 1 LEADER]: Becoming Leader. State: Replica: 557c40d87c194c8889eec2e653935455, State: Running, Role: LEADER
I20260620 12:13:49.781126 27389 consensus_queue.cc:237] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:49.782006 27174 raft_consensus.cc:2454] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 1.
I20260620 12:13:49.784152 26875 catalog_manager.cc:5495] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 557c40d87c194c8889eec2e653935455 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "557c40d87c194c8889eec2e653935455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } health_report { overall_health: HEALTHY } } }
I20260620 12:13:49.807515 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.803560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.24.141.3:34153)
0620 12:13:49.803683 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.803688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.803702 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:13:49.803942 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.803944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.804016 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.804130 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:49.804394 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.804942 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.805869 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.806093 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.806181 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.806449 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.807092 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.807204 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.807261 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.807343 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:13:49.824201 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.818230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:51407 (local address 127.24.141.2:36931)
0620 12:13:49.818381 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.818385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.818856 (+ 471us) server_negotiation.cc:407] Connection header received
0620 12:13:49.819092 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.819095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.819160 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.819255 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:49.820130 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.820737 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.822234 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.822633 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.822752 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.823351 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.823485 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.823620 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.823898 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.823991 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:13:49.824188 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.818116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.24.141.3:51407)
0620 12:13:49.818712 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.818731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.818893 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.819310 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.819314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.819325 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:49.819946 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.819960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.820905 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.820909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.822009 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.822025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.822910 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.822914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.823026 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.823108 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.823682 (+ 574us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.823692 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.823793 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.823915 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":167,"threads_started":1}
I20260620 12:13:49.832898 27034 raft_consensus.cc:1261] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Refusing update from remote peer 557c40d87c194c8889eec2e653935455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:49.833040 27174 raft_consensus.cc:1261] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Refusing update from remote peer 557c40d87c194c8889eec2e653935455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:49.833666 27440 consensus_queue.cc:1035] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:49.833882 27389 consensus_queue.cc:1035] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:49.858450 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.853755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34153 (local address 127.24.141.2:60781)
0620 12:13:49.853949 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:49.853966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:49.854046 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:49.854483 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:49.854487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:49.854497 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:49.854624 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.854630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.855722 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.855727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.856811 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:49.856825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.857590 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.857594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:49.857700 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.857740 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.858056 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.858061 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:13:49.858116 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 12:13:49.858245 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 12:13:49.858901 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:49.853843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:60781 (local address 127.24.141.3:34153)
0620 12:13:49.853990 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:49.853995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:49.854011 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:13:49.854178 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:49.854181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:49.854240 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:49.854324 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:49.854732 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.855580 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.856969 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:49.857383 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:49.857494 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:49.857860 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:49.857971 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:49.858023 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:49.858691 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 12:13:49.858759 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 12:13:49.859797 27414 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 12:13:51.432924 26875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:51.433183 26875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:51.436681 26875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Sending DeleteTablet for 3 replicas of tablet 55fcca1aa0d340babee0158f47d74eac
I20260620 12:13:51.437466 27154 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fcca1aa0d340babee0158f47d74eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:51 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:53302
I20260620 12:13:51.437534 27014 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fcca1aa0d340babee0158f47d74eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:51 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:37216
I20260620 12:13:51.437553 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fcca1aa0d340babee0158f47d74eac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:51 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45558
I20260620 12:13:51.437906 27470 tablet_replica.cc:330] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: stopping tablet replica
I20260620 12:13:51.438052 27470 raft_consensus.cc:2229] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:51.438061 27471 tablet_replica.cc:330] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: stopping tablet replica
I20260620 12:13:51.438190 27470 raft_consensus.cc:2258] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:51.438215 27471 raft_consensus.cc:2229] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:51.438367 27471 raft_consensus.cc:2258] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:51.438202 27472 tablet_replica.cc:330] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: stopping tablet replica
I20260620 12:13:51.438752 27472 raft_consensus.cc:2229] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:13:51.439133 27470 ts_tablet_manager.cc:1905] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:51.439186 27472 raft_consensus.cc:2258] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:51.440814 27472 ts_tablet_manager.cc:1905] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:51.442258 27471 ts_tablet_manager.cc:1905] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:51.443151 27470 ts_tablet_manager.cc:1918] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:13:51.443296 27470 log.cc:1196] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/55fcca1aa0d340babee0158f47d74eac
I20260620 12:13:51.443559 27470 ts_tablet_manager.cc:1939] T 55fcca1aa0d340babee0158f47d74eac P 07d1a158beab4738bc511c65580cff0d: Deleting consensus metadata
I20260620 12:13:51.444079 27472 ts_tablet_manager.cc:1918] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:13:51.444160 27472 log.cc:1196] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/55fcca1aa0d340babee0158f47d74eac
I20260620 12:13:51.444461 27472 ts_tablet_manager.cc:1939] T 55fcca1aa0d340babee0158f47d74eac P 557c40d87c194c8889eec2e653935455: Deleting consensus metadata
I20260620 12:13:51.444599 26863 catalog_manager.cc:4841] TS 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697): tablet 55fcca1aa0d340babee0158f47d74eac (table test-table [id=99cddfae2b62457d8550ca10321131dc]) successfully deleted
I20260620 12:13:51.445194 26862 catalog_manager.cc:4841] TS 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153): tablet 55fcca1aa0d340babee0158f47d74eac (table test-table [id=99cddfae2b62457d8550ca10321131dc]) successfully deleted
I20260620 12:13:51.445876 27471 ts_tablet_manager.cc:1918] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:13:51.445997 27471 log.cc:1196] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/55fcca1aa0d340babee0158f47d74eac
I20260620 12:13:51.446291 27471 ts_tablet_manager.cc:1939] T 55fcca1aa0d340babee0158f47d74eac P 5618041561a1491d8c4c2e67e77cf55f: Deleting consensus metadata
I20260620 12:13:51.447121 26861 catalog_manager.cc:4841] TS 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931): tablet 55fcca1aa0d340babee0158f47d74eac (table test-table [id=99cddfae2b62457d8550ca10321131dc]) successfully deleted
Jun 20 12:13:51 dist-test-slave-rh5v krb5kdc[26832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:13:51 dist-test-slave-rh5v krb5kdc[26832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:51.478649 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:51.460628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.24.141.62:38009)
0620 12:13:51.467823 (+ 7195us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:51.467832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:51.467853 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:13:51.467905 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:51.467908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:51.467961 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:51.468064 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:51.469500 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.470353 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.471651 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.472020 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.472169 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:51.475498 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:51.475502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:51.475506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:51.475548 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:51.476936 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:51.477460 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:51.477463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:51.477465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:51.477569 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:51.477806 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:51.477808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:51.477809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:51.478003 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:51.478113 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:51.478329 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 12:13:51.478460 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7070,"thread_start_us":114,"threads_started":1}
I20260620 12:13:51.481789 26875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:13:51.482319 26875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:51.488245 27014 tablet_service.cc:1467] Processing CreateTablet for tablet fde7957a2218476cb6fc66a0ba451693 (DEFAULT_TABLE table=test-table [id=4707222704dc4d72b434cfe3a1d265c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:51.488245 27154 tablet_service.cc:1467] Processing CreateTablet for tablet fde7957a2218476cb6fc66a0ba451693 (DEFAULT_TABLE table=test-table [id=4707222704dc4d72b434cfe3a1d265c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:51.488591 27154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde7957a2218476cb6fc66a0ba451693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:51.488591 27014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde7957a2218476cb6fc66a0ba451693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:51.489602 27291 tablet_service.cc:1467] Processing CreateTablet for tablet fde7957a2218476cb6fc66a0ba451693 (DEFAULT_TABLE table=test-table [id=4707222704dc4d72b434cfe3a1d265c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:51.489949 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde7957a2218476cb6fc66a0ba451693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:51.491590 27481 tablet_bootstrap.cc:492] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: Bootstrap starting.
I20260620 12:13:51.492437 27481 tablet_bootstrap.cc:654] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:51.492766 27483 tablet_bootstrap.cc:492] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: Bootstrap starting.
I20260620 12:13:51.493194 27482 tablet_bootstrap.cc:492] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: Bootstrap starting.
I20260620 12:13:51.493796 27483 tablet_bootstrap.cc:654] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:51.494115 27482 tablet_bootstrap.cc:654] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:51.495054 27481 tablet_bootstrap.cc:492] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: No bootstrap required, opened a new log
I20260620 12:13:51.495065 27483 tablet_bootstrap.cc:492] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: No bootstrap required, opened a new log
I20260620 12:13:51.495142 27481 ts_tablet_manager.cc:1397] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 12:13:51.495175 27483 ts_tablet_manager.cc:1397] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:13:51.495568 27482 tablet_bootstrap.cc:492] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: No bootstrap required, opened a new log
I20260620 12:13:51.495536 27481 raft_consensus.cc:348] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.495632 27481 raft_consensus.cc:374] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:51.495676 27481 raft_consensus.cc:729] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5618041561a1491d8c4c2e67e77cf55f, State: Initialized, Role: FOLLOWER
I20260620 12:13:51.495699 27483 raft_consensus.cc:348] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.495826 27483 raft_consensus.cc:374] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:51.495796 27481 consensus_queue.cc:260] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.495863 27483 raft_consensus.cc:729] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557c40d87c194c8889eec2e653935455, State: Initialized, Role: FOLLOWER
I20260620 12:13:51.495965 27482 ts_tablet_manager.cc:1397] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:13:51.496070 27481 ts_tablet_manager.cc:1428] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:13:51.496042 27483 consensus_queue.cc:260] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.496565 27482 raft_consensus.cc:348] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.496728 27483 ts_tablet_manager.cc:1428] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:13:51.497102 27482 raft_consensus.cc:374] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:51.497205 27482 raft_consensus.cc:729] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d1a158beab4738bc511c65580cff0d, State: Initialized, Role: FOLLOWER
I20260620 12:13:51.497395 27482 consensus_queue.cc:260] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.497844 27482 ts_tablet_manager.cc:1428] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 12:13:51.759356 27456 raft_consensus.cc:482] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:51.759533 27456 raft_consensus.cc:504] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.760015 27456 leader_election.cc:288] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153)
I20260620 12:13:51.760695 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde7957a2218476cb6fc66a0ba451693" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d" is_pre_election: true
I20260620 12:13:51.760700 27314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde7957a2218476cb6fc66a0ba451693" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557c40d87c194c8889eec2e653935455" is_pre_election: true
I20260620 12:13:51.760877 27314 raft_consensus.cc:2454] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 0.
I20260620 12:13:51.760876 27034 raft_consensus.cc:2454] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 0.
I20260620 12:13:51.761346 27107 leader_election.cc:302] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 5618041561a1491d8c4c2e67e77cf55f; no voters:
I20260620 12:13:51.761579 27456 raft_consensus.cc:2781] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:51.761699 27456 raft_consensus.cc:482] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:51.761727 27456 raft_consensus.cc:3037] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:51.763065 27456 raft_consensus.cc:504] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.763504 27456 leader_election.cc:288] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 election: Requested vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153)
I20260620 12:13:51.764007 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde7957a2218476cb6fc66a0ba451693" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d"
I20260620 12:13:51.764007 27314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde7957a2218476cb6fc66a0ba451693" candidate_uuid: "5618041561a1491d8c4c2e67e77cf55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557c40d87c194c8889eec2e653935455"
I20260620 12:13:51.764171 27314 raft_consensus.cc:3037] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:51.764171 27034 raft_consensus.cc:3037] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:51.765511 27314 raft_consensus.cc:2454] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 1.
I20260620 12:13:51.765511 27034 raft_consensus.cc:2454] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5618041561a1491d8c4c2e67e77cf55f in term 1.
I20260620 12:13:51.765914 27106 leader_election.cc:302] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 557c40d87c194c8889eec2e653935455, 5618041561a1491d8c4c2e67e77cf55f; no voters:
I20260620 12:13:51.766184 27456 raft_consensus.cc:2781] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:51.766290 27456 raft_consensus.cc:686] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 1 LEADER]: Becoming Leader. State: Replica: 5618041561a1491d8c4c2e67e77cf55f, State: Running, Role: LEADER
I20260620 12:13:51.766492 27456 consensus_queue.cc:237] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.768674 26873 catalog_manager.cc:5495] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2). New cstate: current_term: 1 leader_uuid: "5618041561a1491d8c4c2e67e77cf55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } health_report { overall_health: UNKNOWN } } }
I20260620 12:13:51.801723 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:51.798193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.24.141.2:36931)
0620 12:13:51.798494 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:51.798501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:51.798513 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:13:51.798577 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:51.798580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:51.798627 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:51.798705 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:51.799015 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.799719 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.800619 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.800853 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.800947 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:51.801245 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:51.801333 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:51.801391 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:51.801486 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 12:13:51.801540 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260620 12:13:51.813274 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:51.809672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:32925 (local address 127.24.141.2:36931)
0620 12:13:51.809841 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:51.809845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:51.809974 (+ 129us) server_negotiation.cc:407] Connection header received
0620 12:13:51.810044 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:51.810047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:51.810122 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:51.810241 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:51.810529 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.811087 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.811952 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.812219 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.812328 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:51.812636 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:51.812744 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:51.812819 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:51.812977 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 12:13:51.813053 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":67}
I20260620 12:13:51.813283 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:51.809449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.24.141.2:32925)
0620 12:13:51.809813 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:51.809826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:51.809964 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:51.810259 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:51.810261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:51.810268 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:13:51.810394 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:51.810399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.811198 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.811200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:51.811826 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:51.811832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:51.812327 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:51.812330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:51.812428 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:51.812458 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:13:51.812873 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:13:51.812882 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:13:51.812955 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 12:13:51.813045 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":7,"thread_start_us":106,"threads_started":1}
I20260620 12:13:51.819698 27034 raft_consensus.cc:1261] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Refusing update from remote peer 5618041561a1491d8c4c2e67e77cf55f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:51.819746 27314 raft_consensus.cc:1261] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Refusing update from remote peer 5618041561a1491d8c4c2e67e77cf55f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:51.820087 27390 consensus_queue.cc:1035] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:51.820292 27451 consensus_queue.cc:1035] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:51.858371 26875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:51.858548 26875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:51.861100 26875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Sending DeleteTablet for 3 replicas of tablet fde7957a2218476cb6fc66a0ba451693
I20260620 12:13:51.861867 27014 tablet_service.cc:1514] Processing DeleteTablet for tablet fde7957a2218476cb6fc66a0ba451693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:51 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:37216
I20260620 12:13:51.861965 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet fde7957a2218476cb6fc66a0ba451693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:51 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45558
I20260620 12:13:51.862133 27470 tablet_replica.cc:330] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: stopping tablet replica
I20260620 12:13:51.862237 27472 tablet_replica.cc:330] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: stopping tablet replica
I20260620 12:13:51.862298 27470 raft_consensus.cc:2229] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:51.862411 27472 raft_consensus.cc:2229] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:51.862478 27470 raft_consensus.cc:2258] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:51.862586 27472 raft_consensus.cc:2258] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:51.862773 27154 tablet_service.cc:1514] Processing DeleteTablet for tablet fde7957a2218476cb6fc66a0ba451693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:51 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:53302
I20260620 12:13:51.862995 27471 tablet_replica.cc:330] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: stopping tablet replica
I20260620 12:13:51.863157 27471 raft_consensus.cc:2229] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:13:51.863577 27471 raft_consensus.cc:2258] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:51.863653 27470 ts_tablet_manager.cc:1905] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:51.864496 27471 ts_tablet_manager.cc:1905] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:51.866355 27470 ts_tablet_manager.cc:1918] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:51.866460 27470 log.cc:1196] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/fde7957a2218476cb6fc66a0ba451693
I20260620 12:13:51.866750 27470 ts_tablet_manager.cc:1939] T fde7957a2218476cb6fc66a0ba451693 P 07d1a158beab4738bc511c65580cff0d: Deleting consensus metadata
I20260620 12:13:51.867673 27471 ts_tablet_manager.cc:1918] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:51.867766 27471 log.cc:1196] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/fde7957a2218476cb6fc66a0ba451693
I20260620 12:13:51.868017 27471 ts_tablet_manager.cc:1939] T fde7957a2218476cb6fc66a0ba451693 P 5618041561a1491d8c4c2e67e77cf55f: Deleting consensus metadata
I20260620 12:13:51.868086 26863 catalog_manager.cc:4841] TS 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697): tablet fde7957a2218476cb6fc66a0ba451693 (table test-table [id=4707222704dc4d72b434cfe3a1d265c5]) successfully deleted
I20260620 12:13:51.868839 26861 catalog_manager.cc:4841] TS 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931): tablet fde7957a2218476cb6fc66a0ba451693 (table test-table [id=4707222704dc4d72b434cfe3a1d265c5]) successfully deleted
I20260620 12:13:51.869194 26875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:13:51.869537 27472 ts_tablet_manager.cc:1905] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 12:13:51.869706 26875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:13:51.872606 27472 ts_tablet_manager.cc:1918] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:13:51.872711 27472 log.cc:1196] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/fde7957a2218476cb6fc66a0ba451693
I20260620 12:13:51.872967 27472 ts_tablet_manager.cc:1939] T fde7957a2218476cb6fc66a0ba451693 P 557c40d87c194c8889eec2e653935455: Deleting consensus metadata
I20260620 12:13:51.873971 26862 catalog_manager.cc:4841] TS 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153): tablet fde7957a2218476cb6fc66a0ba451693 (table test-table [id=4707222704dc4d72b434cfe3a1d265c5]) successfully deleted
I20260620 12:13:51.877790 27014 tablet_service.cc:1467] Processing CreateTablet for tablet a00b617f0eaa403397af29a89acd0341 (DEFAULT_TABLE table=test-table [id=3dce21050dde41a2aae3ecc40f23b1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:51.877795 27291 tablet_service.cc:1467] Processing CreateTablet for tablet a00b617f0eaa403397af29a89acd0341 (DEFAULT_TABLE table=test-table [id=3dce21050dde41a2aae3ecc40f23b1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:51.878172 27014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00b617f0eaa403397af29a89acd0341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:51.878173 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00b617f0eaa403397af29a89acd0341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:51.880975 27482 tablet_bootstrap.cc:492] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d: Bootstrap starting.
I20260620 12:13:51.881606 27483 tablet_bootstrap.cc:492] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: Bootstrap starting.
I20260620 12:13:51.882022 27482 tablet_bootstrap.cc:654] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:51.882651 27483 tablet_bootstrap.cc:654] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:51.883134 27154 tablet_service.cc:1467] Processing CreateTablet for tablet a00b617f0eaa403397af29a89acd0341 (DEFAULT_TABLE table=test-table [id=3dce21050dde41a2aae3ecc40f23b1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:13:51.883459 27154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00b617f0eaa403397af29a89acd0341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:51.884934 27482 tablet_bootstrap.cc:492] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d: No bootstrap required, opened a new log
I20260620 12:13:51.885054 27482 ts_tablet_manager.cc:1397] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 12:13:51.885573 27482 raft_consensus.cc:348] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.885713 27482 raft_consensus.cc:374] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:51.885746 27482 raft_consensus.cc:729] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d1a158beab4738bc511c65580cff0d, State: Initialized, Role: FOLLOWER
I20260620 12:13:51.885891 27482 consensus_queue.cc:260] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.886157 27482 ts_tablet_manager.cc:1428] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:13:51.886399 27481 tablet_bootstrap.cc:492] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f: Bootstrap starting.
I20260620 12:13:51.887342 27481 tablet_bootstrap.cc:654] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:51.890127 27481 tablet_bootstrap.cc:492] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f: No bootstrap required, opened a new log
I20260620 12:13:51.890128 27483 tablet_bootstrap.cc:492] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: No bootstrap required, opened a new log
I20260620 12:13:51.890254 27483 ts_tablet_manager.cc:1397] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 12:13:51.890244 27481 ts_tablet_manager.cc:1397] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 12:13:51.890776 27481 raft_consensus.cc:348] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.890799 27483 raft_consensus.cc:348] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.890892 27481 raft_consensus.cc:374] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:51.890928 27481 raft_consensus.cc:729] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5618041561a1491d8c4c2e67e77cf55f, State: Initialized, Role: FOLLOWER
I20260620 12:13:51.890928 27483 raft_consensus.cc:374] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:51.890971 27483 raft_consensus.cc:729] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557c40d87c194c8889eec2e653935455, State: Initialized, Role: FOLLOWER
I20260620 12:13:51.891062 27481 consensus_queue.cc:260] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.891110 27483 consensus_queue.cc:260] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:51.891398 27483 ts_tablet_manager.cc:1428] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:13:51.891397 27481 ts_tablet_manager.cc:1428] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:13:52.057540 27464 raft_consensus.cc:482] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:13:52.057750 27464 raft_consensus.cc:504] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:52.058339 27464 leader_election.cc:288] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931)
I20260620 12:13:52.058933 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00b617f0eaa403397af29a89acd0341" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d" is_pre_election: true
I20260620 12:13:52.058926 27174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00b617f0eaa403397af29a89acd0341" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5618041561a1491d8c4c2e67e77cf55f" is_pre_election: true
I20260620 12:13:52.059095 27034 raft_consensus.cc:2454] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 0.
I20260620 12:13:52.059129 27174 raft_consensus.cc:2454] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 0.
I20260620 12:13:52.059535 27247 leader_election.cc:302] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 557c40d87c194c8889eec2e653935455; no voters:
I20260620 12:13:52.059727 27464 raft_consensus.cc:2781] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:13:52.059794 27464 raft_consensus.cc:482] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:13:52.059834 27464 raft_consensus.cc:3037] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:52.061094 27464 raft_consensus.cc:504] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:52.061465 27464 leader_election.cc:288] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 election: Requested vote from peers 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697), 5618041561a1491d8c4c2e67e77cf55f (127.24.141.2:36931)
I20260620 12:13:52.061928 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00b617f0eaa403397af29a89acd0341" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d1a158beab4738bc511c65580cff0d"
I20260620 12:13:52.062002 27174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00b617f0eaa403397af29a89acd0341" candidate_uuid: "557c40d87c194c8889eec2e653935455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5618041561a1491d8c4c2e67e77cf55f"
I20260620 12:13:52.062140 27174 raft_consensus.cc:3037] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:52.062141 27034 raft_consensus.cc:3037] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:52.063510 27034 raft_consensus.cc:2454] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 1.
I20260620 12:13:52.063510 27174 raft_consensus.cc:2454] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557c40d87c194c8889eec2e653935455 in term 1.
I20260620 12:13:52.063932 27247 leader_election.cc:302] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d1a158beab4738bc511c65580cff0d, 557c40d87c194c8889eec2e653935455; no voters:
I20260620 12:13:52.064224 27464 raft_consensus.cc:2781] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:52.064441 27464 raft_consensus.cc:686] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 1 LEADER]: Becoming Leader. State: Replica: 557c40d87c194c8889eec2e653935455, State: Running, Role: LEADER
I20260620 12:13:52.064635 27464 consensus_queue.cc:237] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } }
I20260620 12:13:52.066684 26877 catalog_manager.cc:5495] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 557c40d87c194c8889eec2e653935455 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "557c40d87c194c8889eec2e653935455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557c40d87c194c8889eec2e653935455" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34153 } health_report { overall_health: HEALTHY } } }
I20260620 12:13:52.096697 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:52.093418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.24.141.3:34153)
0620 12:13:52.093770 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:52.093792 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:52.093805 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:13:52.093850 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:52.093852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:52.093897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:52.093972 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:13:52.094242 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:52.094785 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:52.095624 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:52.095855 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:52.095944 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:52.096200 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:13:52.096297 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:13:52.096358 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:52.096456 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 12:13:52.096516 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":114,"threads_started":1}
I20260620 12:13:52.105648 27034 raft_consensus.cc:1261] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d [term 1 FOLLOWER]: Refusing update from remote peer 557c40d87c194c8889eec2e653935455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:52.105805 27174 raft_consensus.cc:1261] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Refusing update from remote peer 557c40d87c194c8889eec2e653935455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:13:52.106137 27500 consensus_queue.cc:1035] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d1a158beab4738bc511c65580cff0d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:13:52.106346 27464 consensus_queue.cc:1035] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5618041561a1491d8c4c2e67e77cf55f" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:13:52.128784 27453 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 12:13:53.695340 26873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:53.695535 26873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:13:53.698755 26873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6962a35802984f52ae28cbb04d4304d3: Sending DeleteTablet for 3 replicas of tablet a00b617f0eaa403397af29a89acd0341
I20260620 12:13:53.699560 27154 tablet_service.cc:1514] Processing DeleteTablet for tablet a00b617f0eaa403397af29a89acd0341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:53 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:53302
I20260620 12:13:53.699560 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet a00b617f0eaa403397af29a89acd0341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:53 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45558
I20260620 12:13:53.699752 27014 tablet_service.cc:1514] Processing DeleteTablet for tablet a00b617f0eaa403397af29a89acd0341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:13:53 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:37216
I20260620 12:13:53.700057 27528 tablet_replica.cc:330] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f: stopping tablet replica
I20260620 12:13:53.700253 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 26949
I20260620 12:13:53.700270 27530 tablet_replica.cc:330] T a00b617f0eaa403397af29a89acd0341 P 07d1a158beab4738bc511c65580cff0d: stopping tablet replica
I20260620 12:13:53.700259 27528 raft_consensus.cc:2229] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:13:53.701184 27528 raft_consensus.cc:2258] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:53.700335 27529 tablet_replica.cc:330] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: stopping tablet replica
I20260620 12:13:53.702724 27529 raft_consensus.cc:2229] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:13:53.703166 27529 raft_consensus.cc:2258] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:13:53.704187 27529 ts_tablet_manager.cc:1905] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:53.707152 27529 ts_tablet_manager.cc:1918] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:13:53.707239 27529 log.cc:1196] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/a00b617f0eaa403397af29a89acd0341
I20260620 12:13:53.707592 27529 ts_tablet_manager.cc:1939] T a00b617f0eaa403397af29a89acd0341 P 557c40d87c194c8889eec2e653935455: Deleting consensus metadata
I20260620 12:13:53.708047 27528 ts_tablet_manager.cc:1905] T a00b617f0eaa403397af29a89acd0341 P 5618041561a1491d8c4c2e67e77cf55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:13:53.708613 26862 catalog_manager.cc:4841] TS 557c40d87c194c8889eec2e653935455 (127.24.141.3:34153): tablet a00b617f0eaa403397af29a89acd0341 (table test-table [id=3dce21050dde41a2aae3ecc40f23b1eb]) successfully deleted
W20260620 12:13:53.709201 26863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:45697 (error 108)
I20260620 12:13:53.709426 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 27089
I20260620 12:13:53.711978 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:53.711305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45697 (local address 127.0.0.1:37232)
0620 12:13:53.711711 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:53.711848 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
W20260620 12:13:53.712452 26863 catalog_manager.cc:4584] TS 07d1a158beab4738bc511c65580cff0d (127.24.141.1:45697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a00b617f0eaa403397af29a89acd0341: Network error: Client connection negotiation failed: client connection to 127.24.141.1:45697: connect: Connection refused (error 111)
I20260620 12:13:53.716922 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 27229
I20260620 12:13:53.718765 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:53.718201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:36931 (local address 127.0.0.1:53334)
0620 12:13:53.718601 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:53.718656 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.2:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260620 12:13:53.723744 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 26845
2026-06-20T12:13:53Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6803 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:53 dist-test-slave-rh5v krb5kdc[27535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:53 dist-test-slave-rh5v krb5kdc[27535](info): set up 2 sockets
Jun 20 12:13:53 dist-test-slave-rh5v krb5kdc[27535](info): commencing operation
krb5kdc: starting...
W20260620 12:13:54.685735 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:13:54 dist-test-slave-rh5v krb5kdc[27535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:13:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:13:54.820017 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:46597
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:54.954236 27548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:54.954625 27548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:54.954672 27548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:54.959069 27548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:13:54.959167 27548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:54.959187 27548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:54.959204 27548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:13:54.959219 27548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:13:54.965783 27548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:54.967337 27548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:54.968714 27548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:54.979825 27556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:54.979951 27553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:54.979856 27554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:54.980217 27548 server_base.cc:1034] running on GCE node
I20260620 12:13:54.981041 27548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:54.982721 27548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:54.983979 27548 hybrid_clock.cc:648] HybridClock initialized: now 1781957634983963 us; error 61 us; skew 500 ppm
Jun 20 12:13:54 dist-test-slave-rh5v krb5kdc[27535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:54.987772 27548 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:13:54.989121 27548 webserver.cc:460] Webserver started at http://127.24.141.62:40075/ using document root <none> and password file <none>
I20260620 12:13:54.989873 27548 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:54.989969 27548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:54.990234 27548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:54.992959 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "b3428a5eac5b4818b08a1028144c58dc"
format_stamp: "Formatted at 2026-06-20 12:13:54 on dist-test-slave-rh5v"
server_key: "51db38c83d39f426f09b643618c5f8f3"
server_key_iv: "b2f1444a525ee1501f026a826c72e197"
server_key_version: "encryptionkey@0"
I20260620 12:13:54.993763 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "b3428a5eac5b4818b08a1028144c58dc"
format_stamp: "Formatted at 2026-06-20 12:13:54 on dist-test-slave-rh5v"
server_key: "51db38c83d39f426f09b643618c5f8f3"
server_key_iv: "b2f1444a525ee1501f026a826c72e197"
server_key_version: "encryptionkey@0"
I20260620 12:13:54.999032 27548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:13:55.002739 27563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.004623 27548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:13:55.004877 27548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "b3428a5eac5b4818b08a1028144c58dc"
format_stamp: "Formatted at 2026-06-20 12:13:54 on dist-test-slave-rh5v"
server_key: "51db38c83d39f426f09b643618c5f8f3"
server_key_iv: "b2f1444a525ee1501f026a826c72e197"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.005043 27548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:55.039268 27548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:55.043146 27548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:55.043372 27548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:55.052294 27548 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:46597
I20260620 12:13:55.052384 27615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:46597 every 8 connection(s)
I20260620 12:13:55.053644 27548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:13:55.057967 27616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:55.059340 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 27548
I20260620 12:13:55.059468 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:13:55.059922 25140 external_mini_cluster.cc:1428] Setting key 7bf112e21713de0cdab14e1c32efd2d9
I20260620 12:13:55.065760 27616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc: Bootstrap starting.
I20260620 12:13:55.068511 27616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:55.069499 27616 log.cc:824] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:13:55 dist-test-slave-rh5v krb5kdc[27535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:55.072647 27616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc: No bootstrap required, opened a new log
I20260620 12:13:55.078362 27616 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3428a5eac5b4818b08a1028144c58dc" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46597 } }
I20260620 12:13:55.078739 27616 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:55.078791 27616 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3428a5eac5b4818b08a1028144c58dc, State: Initialized, Role: FOLLOWER
I20260620 12:13:55.079039 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:55.062002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.24.141.62:46597)
0620 12:13:55.062625 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:55.062653 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:55.062703 (+ 50us) server_negotiation.cc:407] Connection header received
0620 12:13:55.063617 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:55.063641 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:55.064119 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:55.064533 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:55.065543 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.066732 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.067679 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.068270 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.068439 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:55.071743 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:55.071752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:55.071776 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:55.071818 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:55.074243 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.074939 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.074948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.074956 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.075133 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.075745 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.075750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.075753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.076511 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:55.076775 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:55.077262 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 12:13:55.077727 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":195,"threads_started":1}
I20260620 12:13:55.079432 27616 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3428a5eac5b4818b08a1028144c58dc" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46597 } }
I20260620 12:13:55.079622 27616 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:13:55.079691 27616 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:13:55.079805 27616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:55.081192 27616 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3428a5eac5b4818b08a1028144c58dc" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46597 } }
I20260620 12:13:55.081641 27616 leader_election.cc:302] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3428a5eac5b4818b08a1028144c58dc; no voters:
I20260620 12:13:55.082031 27616 leader_election.cc:288] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:13:55.082152 27621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:55.082475 27621 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [term 1 LEADER]: Becoming Leader. State: Replica: b3428a5eac5b4818b08a1028144c58dc, State: Running, Role: LEADER
I20260620 12:13:55.082939 27621 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3428a5eac5b4818b08a1028144c58dc" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46597 } }
I20260620 12:13:55.083885 27616 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:13:55.085032 27622 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3428a5eac5b4818b08a1028144c58dc. Latest consensus state: current_term: 1 leader_uuid: "b3428a5eac5b4818b08a1028144c58dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3428a5eac5b4818b08a1028144c58dc" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46597 } } }
I20260620 12:13:55.085314 27622 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:55.085525 27623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3428a5eac5b4818b08a1028144c58dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3428a5eac5b4818b08a1028144c58dc" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46597 } } }
I20260620 12:13:55.085758 27623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc [sys.catalog]: This master's current role is: LEADER
W20260620 12:13:55.090240 27636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:13:55.090363 27636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:13:55.090492 27631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:13:55.091384 27631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:13:55.100353 27631 catalog_manager.cc:1347] Generated new cluster ID: 73d9827e96e142b0b2bd1b4decca24d8
I20260620 12:13:55.100525 27631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:13:55.125514 27631 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:13:55.126652 27631 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:13:55.132299 27631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3428a5eac5b4818b08a1028144c58dc: Generated new TSK 0
I20260620 12:13:55.133605 27631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:13:55.172768 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:46597
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:55.314772 27642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:55.315128 27642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:55.315199 27642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:55.319667 27642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:55.319773 27642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:55.319875 27642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:13:55.326882 27642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:55.328171 27642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:55.329408 27642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:13:55.343530 27642 server_base.cc:1034] running on GCE node
W20260620 12:13:55.343482 27648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:55.343458 27650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:55.343458 27647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:55.344376 27642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:55.345293 27642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:55.346568 27642 hybrid_clock.cc:648] HybridClock initialized: now 1781957635346535 us; error 52 us; skew 500 ppm
Jun 20 12:13:55 dist-test-slave-rh5v krb5kdc[27535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:55.350941 27642 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:13:55.352440 27642 webserver.cc:460] Webserver started at http://127.24.141.1:37873/ using document root <none> and password file <none>
I20260620 12:13:55.353229 27642 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:55.353338 27642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:55.353571 27642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:55.355949 27642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "67344ed42c7146c987a269c1085cf627"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "07a8a03261cb22039c186b4c9d1ef77b"
server_key_iv: "0de9412e08ee80c0f18a3dfa1adc0fa1"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.356801 27642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "67344ed42c7146c987a269c1085cf627"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "07a8a03261cb22039c186b4c9d1ef77b"
server_key_iv: "0de9412e08ee80c0f18a3dfa1adc0fa1"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.362105 27642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:13:55.365886 27657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.367731 27642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:13:55.367934 27642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "67344ed42c7146c987a269c1085cf627"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "07a8a03261cb22039c186b4c9d1ef77b"
server_key_iv: "0de9412e08ee80c0f18a3dfa1adc0fa1"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.368095 27642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:55.381019 27642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:55.384469 27642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:55.384724 27642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:55.385465 27642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:55.386494 27642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:55.386550 27642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.386598 27642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:55.386631 27642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.396695 27642 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:34917
I20260620 12:13:55.396780 27770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:34917 every 8 connection(s)
I20260620 12:13:55.397820 27642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:13:55.402710 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 27642
I20260620 12:13:55.402868 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:13:55.403184 25140 external_mini_cluster.cc:1428] Setting key 2d828a184be10829b6324166b734dd51
Jun 20 12:13:55 dist-test-slave-rh5v krb5kdc[27535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:13:55.417802 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:55.400463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:50623 (local address 127.24.141.62:46597)
0620 12:13:55.400634 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:55.400638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:55.402014 (+ 1376us) server_negotiation.cc:407] Connection header received
0620 12:13:55.403454 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:55.403458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:55.403540 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:55.403680 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:55.405428 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.406007 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.407865 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.408144 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.408254 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:55.411912 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:55.411918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:55.411922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:55.411978 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:55.414015 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.414896 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.414901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.414904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.415018 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.415747 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.415750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.415752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.415986 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:55.416293 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:55.417441 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0620 12:13:55.417642 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:13:55.418850 27773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:55.400908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:46597 (local address 127.24.141.1:50623)
0620 12:13:55.401803 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:55.401869 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:55.403080 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:55.403914 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:55.403924 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:55.404436 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:55.405234 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:55.405261 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.406184 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.406189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:55.407700 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:55.407715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.408333 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.408338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:55.408558 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:55.409328 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:55.409351 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:55.411565 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:55.414245 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:55.414255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:55.414278 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:55.414717 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:55.415255 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:55.415259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:55.415261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:55.415572 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:55.416392 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:55.416401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:55.416825 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 12:13:55.417145 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 12:13:55.417604 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":202,"threads_started":1}
I20260620 12:13:55.420606 27771 heartbeater.cc:347] Connected to a master server at 127.24.141.62:46597
I20260620 12:13:55.420899 27771 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:55.421586 27771 heartbeater.cc:511] Master 127.24.141.62:46597 requested a full tablet report, sending...
I20260620 12:13:55.423844 27575 ts_manager.cc:194] Registered new tserver with Master: 67344ed42c7146c987a269c1085cf627 (127.24.141.1:34917)
I20260620 12:13:55.425853 27575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:50623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:13:55.434013 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:46597
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:55.571715 27776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:55.572098 27776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:55.572149 27776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:55.576776 27776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:55.576860 27776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:55.576936 27776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:13:55.583096 27776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:55.584743 27776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:55.586277 27776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:55.599874 27784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:55.599833 27782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:55.599833 27781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:55.600121 27776 server_base.cc:1034] running on GCE node
I20260620 12:13:55.600884 27776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:55.601809 27776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:55.603055 27776 hybrid_clock.cc:648] HybridClock initialized: now 1781957635603026 us; error 43 us; skew 500 ppm
Jun 20 12:13:55 dist-test-slave-rh5v krb5kdc[27535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:55.606854 27776 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:13:55.608007 27776 webserver.cc:460] Webserver started at http://127.24.141.2:41587/ using document root <none> and password file <none>
I20260620 12:13:55.608678 27776 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:55.608749 27776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:55.608943 27776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:55.611732 27776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "f11a1c6a014541ac9b9b2fcb96b719dc"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "9c6c52b6f2e08f2f3f447ff01a2d9ab8"
server_key_iv: "8ccd85d6bcea0bb6b39141961b3b552f"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.612396 27776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "f11a1c6a014541ac9b9b2fcb96b719dc"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "9c6c52b6f2e08f2f3f447ff01a2d9ab8"
server_key_iv: "8ccd85d6bcea0bb6b39141961b3b552f"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.617321 27776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:13:55.620707 27791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.622502 27776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:55.622682 27776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "f11a1c6a014541ac9b9b2fcb96b719dc"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "9c6c52b6f2e08f2f3f447ff01a2d9ab8"
server_key_iv: "8ccd85d6bcea0bb6b39141961b3b552f"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.622824 27776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:55.648450 27776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:55.652189 27776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:55.652408 27776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:55.653149 27776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:55.654157 27776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:55.654211 27776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.654256 27776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:55.654280 27776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.664959 27776 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:35891
I20260620 12:13:55.665135 27904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:35891 every 8 connection(s)
I20260620 12:13:55.666711 27776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:13:55.673107 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 27776
I20260620 12:13:55.673229 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:13:55.673552 25140 external_mini_cluster.cc:1428] Setting key b646789cd8caa505156e55da3007b092
Jun 20 12:13:55 dist-test-slave-rh5v krb5kdc[27535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:55.683542 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:55.669454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:46993 (local address 127.24.141.62:46597)
0620 12:13:55.669692 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:55.669697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:55.671139 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 12:13:55.672209 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:55.672213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:55.672286 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:55.672392 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:55.674149 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.674746 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.676201 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.676467 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.676567 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:55.679740 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:55.679744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:55.679747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:55.679775 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:55.681251 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.681859 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.681862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.681863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.681945 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.682298 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.682308 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.682309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.682520 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:55.682612 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:55.683204 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 12:13:55.683364 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 12:13:55.684587 27907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:55.669902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:46597 (local address 127.24.141.2:46993)
0620 12:13:55.670920 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:55.670986 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:55.671941 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:55.672600 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:55.672610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:55.673232 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:55.673924 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:55.673943 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.674917 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.674921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:55.676029 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:55.676043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.676643 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.676647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:55.676839 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:55.677603 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:55.677626 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:55.679513 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:55.681405 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:55.681412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:55.681432 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:55.681742 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:55.682034 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:55.682037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:55.682039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:55.682196 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:55.682627 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:55.682636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:55.682931 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 12:13:55.683123 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 12:13:55.683526 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":185,"threads_started":1}
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:13:55.686197 27905 heartbeater.cc:347] Connected to a master server at 127.24.141.62:46597
I20260620 12:13:55.686596 27905 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:13:55.687374 27905 heartbeater.cc:511] Master 127.24.141.62:46597 requested a full tablet report, sending...
I20260620 12:13:55.688988 27575 ts_manager.cc:194] Registered new tserver with Master: f11a1c6a014541ac9b9b2fcb96b719dc (127.24.141.2:35891)
I20260620 12:13:55.689898 27575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:46993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:13:55.704358 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:46597
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:55.838493 27910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:55.838836 27910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:55.838882 27910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:55.843261 27910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:55.843395 27910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:55.843469 27910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:13:55.850034 27910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:55.851662 27910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:55.853603 27910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:55.867560 27918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:55.867560 27915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:55.867607 27916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:55.868101 27910 server_base.cc:1034] running on GCE node
I20260620 12:13:55.868701 27910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:55.869551 27910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:55.870791 27910 hybrid_clock.cc:648] HybridClock initialized: now 1781957635870747 us; error 66 us; skew 500 ppm
Jun 20 12:13:55 dist-test-slave-rh5v krb5kdc[27535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:55.879702 27910 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:13:55.881043 27910 webserver.cc:460] Webserver started at http://127.24.141.3:39499/ using document root <none> and password file <none>
I20260620 12:13:55.881965 27910 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:55.882061 27910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:55.882321 27910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:55.884989 27910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "bf15d5d96d5142849d013b107f9c50db"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "ac84bc992fe8aff55b8f39bf88af4344"
server_key_iv: "efdb4b43c312db1234e0752576bd9d0c"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.885767 27910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "bf15d5d96d5142849d013b107f9c50db"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "ac84bc992fe8aff55b8f39bf88af4344"
server_key_iv: "efdb4b43c312db1234e0752576bd9d0c"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.890884 27910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 12:13:55.894896 27925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.896755 27910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 12:13:55.896977 27910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "bf15d5d96d5142849d013b107f9c50db"
format_stamp: "Formatted at 2026-06-20 12:13:55 on dist-test-slave-rh5v"
server_key: "ac84bc992fe8aff55b8f39bf88af4344"
server_key_iv: "efdb4b43c312db1234e0752576bd9d0c"
server_key_version: "encryptionkey@0"
I20260620 12:13:55.897184 27910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:55.915311 27910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:55.919148 27910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:55.919368 27910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:55.920087 27910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:55.921167 27910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:55.921223 27910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.921283 27910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:55.921312 27910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:55.931885 27910 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:46435
I20260620 12:13:55.931941 28038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:46435 every 8 connection(s)
I20260620 12:13:55.933003 27910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:13:55.933678 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 27910
I20260620 12:13:55.933795 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:13:55.934074 25140 external_mini_cluster.cc:1428] Setting key 86ae96b305c285df71a51395a285696e
Jun 20 12:13:55 dist-test-slave-rh5v krb5kdc[27535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:55.950605 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:55.935676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:43017 (local address 127.24.141.62:46597)
0620 12:13:55.936795 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:55.936800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:55.937402 (+ 602us) server_negotiation.cc:407] Connection header received
0620 12:13:55.938509 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:55.938513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:55.938575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:55.938652 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:55.940155 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.940748 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.942044 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.942275 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.942372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:55.945761 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:55.945765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:55.945767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:55.945800 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:55.947531 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.948384 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.948388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.948389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.948495 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:55.949009 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:55.949013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:55.949015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:55.949481 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:55.949600 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:55.950252 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 12:13:55.950421 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004}
I20260620 12:13:55.951695 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:55.936271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:46597 (local address 127.24.141.3:43017)
0620 12:13:55.937234 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:55.937290 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:55.938280 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:55.938873 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:55.938881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:55.939367 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:55.940001 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:55.940018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.940870 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.940873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:55.941910 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:55.941931 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:55.942447 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:55.942452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:55.942656 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:55.943412 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:55.943434 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:55.945483 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:55.947732 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:55.947741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:55.947766 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:55.948260 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:55.948627 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:55.948629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:55.948631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:55.948884 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:55.949605 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:55.949614 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:55.949959 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 12:13:55.950173 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 12:13:55.950582 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":167,"threads_started":1}
I20260620 12:13:55.953370 28039 heartbeater.cc:347] Connected to a master server at 127.24.141.62:46597
I20260620 12:13:55.953694 28039 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:55.954327 28039 heartbeater.cc:511] Master 127.24.141.62:46597 requested a full tablet report, sending...
I20260620 12:13:55.955796 27575 ts_manager.cc:194] Registered new tserver with Master: bf15d5d96d5142849d013b107f9c50db (127.24.141.3:46435)
I20260620 12:13:55.956611 27575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:43017
I20260620 12:13:55.962347 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:13:55.979597 27619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:13:55.977950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.24.141.62:46597)
0620 12:13:55.978156 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:55.978161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:55.978283 (+ 122us) server_negotiation.cc:407] Connection header received
0620 12:13:55.978445 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:55.978449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:55.978498 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:55.978569 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:55.979466 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34768: BlockingRecv error: recv got EOF from 127.0.0.1:34768 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:13:55.981640 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 27642
I20260620 12:13:55.988323 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 27776
I20260620 12:13:55.995038 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 27910
I20260620 12:13:56.001184 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 27548
2026-06-20T12:13:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2277 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:56 dist-test-slave-rh5v krb5kdc[28051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:56 dist-test-slave-rh5v krb5kdc[28051](info): set up 2 sockets
Jun 20 12:13:56 dist-test-slave-rh5v krb5kdc[28051](info): commencing operation
krb5kdc: starting...
W20260620 12:13:56.961824 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:13:57 dist-test-slave-rh5v krb5kdc[28051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:13:57.100356 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:35095
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:57.228003 28064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:57.228397 28064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:57.228461 28064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:57.233420 28064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:13:57.233531 28064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:57.233560 28064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:57.233582 28064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:13:57.233601 28064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:13:57.240016 28064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:35095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:57.241617 28064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:57.242981 28064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:57.254211 28072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:57.254205 28069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:57.254597 28064 server_base.cc:1034] running on GCE node
W20260620 12:13:57.254220 28070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:57.255517 28064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:57.256932 28064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:57.258181 28064 hybrid_clock.cc:648] HybridClock initialized: now 1781957637258145 us; error 56 us; skew 500 ppm
Jun 20 12:13:57 dist-test-slave-rh5v krb5kdc[28051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957637, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:57.261807 28064 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:13:57.263010 28064 webserver.cc:460] Webserver started at http://127.24.141.62:38775/ using document root <none> and password file <none>
I20260620 12:13:57.263697 28064 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:57.263782 28064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:57.264004 28064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:57.266521 28064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "1de8bf7f7f4d4c56a35d88161e2bc787"
format_stamp: "Formatted at 2026-06-20 12:13:57 on dist-test-slave-rh5v"
server_key: "8b5e666a9e1aeb647863505f2e5edac6"
server_key_iv: "8630e2efccf672d9a63f8b37fc118b69"
server_key_version: "encryptionkey@0"
I20260620 12:13:57.267189 28064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "1de8bf7f7f4d4c56a35d88161e2bc787"
format_stamp: "Formatted at 2026-06-20 12:13:57 on dist-test-slave-rh5v"
server_key: "8b5e666a9e1aeb647863505f2e5edac6"
server_key_iv: "8630e2efccf672d9a63f8b37fc118b69"
server_key_version: "encryptionkey@0"
I20260620 12:13:57.272101 28064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:13:57.275535 28079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:57.277513 28064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:13:57.277719 28064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "1de8bf7f7f4d4c56a35d88161e2bc787"
format_stamp: "Formatted at 2026-06-20 12:13:57 on dist-test-slave-rh5v"
server_key: "8b5e666a9e1aeb647863505f2e5edac6"
server_key_iv: "8630e2efccf672d9a63f8b37fc118b69"
server_key_version: "encryptionkey@0"
I20260620 12:13:57.277871 28064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:57.302531 28064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:57.306121 28064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:57.306356 28064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:57.314265 28064 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:35095
I20260620 12:13:57.314710 28131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:35095 every 8 connection(s)
I20260620 12:13:57.315431 28064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:13:57.319249 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28064
I20260620 12:13:57.319410 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:13:57.319545 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:57.319770 25140 external_mini_cluster.cc:1428] Setting key a1744c40b430c14e52497a750474f0ec
I20260620 12:13:57.328583 28132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787: Bootstrap starting.
Jun 20 12:13:57 dist-test-slave-rh5v krb5kdc[28051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:57.331955 28132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:57.332821 28132 log.cc:824] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787: Log is configured to *not* fsync() on all Append() calls
I20260620 12:13:57.335542 28132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787: No bootstrap required, opened a new log
I20260620 12:13:57.337893 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:57.321687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.24.141.62:35095)
0620 12:13:57.322257 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:57.322275 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:57.322322 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:13:57.323152 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:57.323175 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:57.323576 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:57.323933 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:57.325141 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:57.326258 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:57.327594 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:57.328190 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:57.328385 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:57.331831 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:57.331840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:57.331862 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:57.331911 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:57.333913 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:57.334496 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:57.334503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:57.334510 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:57.334672 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:57.334973 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:57.334977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:57.334979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:57.335550 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:57.335761 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:57.336119 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 12:13:57.336476 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":168,"threads_started":1}
I20260620 12:13:57.339838 28132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 35095 } }
I20260620 12:13:57.340098 28132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:57.340134 28132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de8bf7f7f4d4c56a35d88161e2bc787, State: Initialized, Role: FOLLOWER
I20260620 12:13:57.340626 28132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 35095 } }
I20260620 12:13:57.340832 28132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:13:57.340907 28132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:13:57.341008 28132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:57.342088 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:13:57.342128 28132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 35095 } }
I20260620 12:13:57.342535 28132 leader_election.cc:302] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de8bf7f7f4d4c56a35d88161e2bc787; no voters:
I20260620 12:13:57.342958 28132 leader_election.cc:288] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:13:57.343149 28137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:57.343647 28137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [term 1 LEADER]: Becoming Leader. State: Replica: 1de8bf7f7f4d4c56a35d88161e2bc787, State: Running, Role: LEADER
I20260620 12:13:57.344148 28137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 35095 } }
I20260620 12:13:57.344539 28132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:13:57.346429 28139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de8bf7f7f4d4c56a35d88161e2bc787. Latest consensus state: current_term: 1 leader_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 35095 } } }
I20260620 12:13:57.346619 28139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:57.347007 28138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de8bf7f7f4d4c56a35d88161e2bc787" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 35095 } } }
I20260620 12:13:57.347470 28152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:13:57.347476 28138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:57.351630 28152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:13:57.359920 28152 catalog_manager.cc:1347] Generated new cluster ID: 9198097db0af4bcd822e6880e13a485a
I20260620 12:13:57.360016 28152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:13:57.365579 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:57.359130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.24.141.62:35095)
0620 12:13:57.359265 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:57.359270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:57.359293 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:13:57.359415 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:57.359419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:57.359483 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:57.359582 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:57.360369 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:57.361239 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:57.362717 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:57.363097 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:57.363191 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:57.364342 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:57.364347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:57.364349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:13:57.365021 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:57.365088 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:13:57.365109 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:57.365196 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:57.365328 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 12:13:57.365402 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:13:57.389714 28152 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:13:57.391013 28152 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:13:57.397480 28152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de8bf7f7f4d4c56a35d88161e2bc787: Generated new TSK 0
I20260620 12:13:57.398790 28152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:13:57.404837 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28064
2026-06-20T12:13:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1400 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:57 dist-test-slave-rh5v krb5kdc[28165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:57 dist-test-slave-rh5v krb5kdc[28165](info): set up 2 sockets
Jun 20 12:13:57 dist-test-slave-rh5v krb5kdc[28165](info): commencing operation
krb5kdc: starting...
W20260620 12:13:58.377585 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:13:58 dist-test-slave-rh5v krb5kdc[28165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:13:58.512970 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33689
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:33689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:58.647285 28178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:58.647676 28178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:58.647742 28178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:58.652264 28178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:13:58.652380 28178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:58.652418 28178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:58.652442 28178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:13:58.652463 28178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:13:58.658799 28178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:33689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:58.660353 28178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:58.661782 28178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:58.672850 28184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:58.672850 28183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:58.672856 28186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:58.673130 28178 server_base.cc:1034] running on GCE node
I20260620 12:13:58.674350 28178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:58.675853 28178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:58.677148 28178 hybrid_clock.cc:648] HybridClock initialized: now 1781957638677109 us; error 56 us; skew 500 ppm
Jun 20 12:13:58 dist-test-slave-rh5v krb5kdc[28165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957638, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:58.680989 28178 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:13:58.682464 28178 webserver.cc:460] Webserver started at http://127.24.141.62:43145/ using document root <none> and password file <none>
I20260620 12:13:58.683179 28178 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:58.683264 28178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:58.683490 28178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:58.686200 28178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "b72a4a41209046e4bdf02cb7747a7326"
format_stamp: "Formatted at 2026-06-20 12:13:58 on dist-test-slave-rh5v"
server_key: "54e64984a042b9d836aa8a256b6e023c"
server_key_iv: "534bea3d97ae7c3a4c1c7fa2799b399a"
server_key_version: "encryptionkey@0"
I20260620 12:13:58.686962 28178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "b72a4a41209046e4bdf02cb7747a7326"
format_stamp: "Formatted at 2026-06-20 12:13:58 on dist-test-slave-rh5v"
server_key: "54e64984a042b9d836aa8a256b6e023c"
server_key_iv: "534bea3d97ae7c3a4c1c7fa2799b399a"
server_key_version: "encryptionkey@0"
I20260620 12:13:58.691836 28178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:13:58.695019 28193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:58.696741 28178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 12:13:58.696935 28178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "b72a4a41209046e4bdf02cb7747a7326"
format_stamp: "Formatted at 2026-06-20 12:13:58 on dist-test-slave-rh5v"
server_key: "54e64984a042b9d836aa8a256b6e023c"
server_key_iv: "534bea3d97ae7c3a4c1c7fa2799b399a"
server_key_version: "encryptionkey@0"
I20260620 12:13:58.697108 28178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:58.723470 28178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:58.727093 28178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:58.727321 28178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:58.736516 28178 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:33689
I20260620 12:13:58.736804 28245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:33689 every 8 connection(s)
I20260620 12:13:58.737809 28178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:13:58.741536 28246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:13:58.742208 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28178
I20260620 12:13:58.742357 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:13:58.742656 25140 external_mini_cluster.cc:1428] Setting key 7ecc63ae8a6893f21c80a00f41442816
I20260620 12:13:58.749185 28246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326: Bootstrap starting.
I20260620 12:13:58.752314 28246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326: Neither blocks nor log segments found. Creating new log.
I20260620 12:13:58.753379 28246 log.cc:824] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:13:58 dist-test-slave-rh5v krb5kdc[28165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:58.756450 28246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326: No bootstrap required, opened a new log
I20260620 12:13:58.761286 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:58.744278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.24.141.62:33689)
0620 12:13:58.744953 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:58.744973 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:58.745027 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:13:58.746009 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:58.746035 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:58.746991 (+ 956us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:58.747784 (+ 793us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:58.748728 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:58.749985 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:58.750963 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:58.751566 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:58.751736 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:58.755113 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:58.755121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:58.755146 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:58.755187 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:58.757474 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:58.758183 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:58.758190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:58.758196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:58.758361 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:58.758707 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:58.758710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:58.758712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:58.759234 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:58.759421 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:58.759831 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 12:13:58.760232 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":176,"threads_started":1}
I20260620 12:13:58.761305 28246 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72a4a41209046e4bdf02cb7747a7326" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33689 } }
I20260620 12:13:58.761735 28246 raft_consensus.cc:374] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:13:58.761798 28246 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72a4a41209046e4bdf02cb7747a7326, State: Initialized, Role: FOLLOWER
I20260620 12:13:58.762348 28246 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72a4a41209046e4bdf02cb7747a7326" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33689 } }
I20260620 12:13:58.762511 28246 raft_consensus.cc:388] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:13:58.762580 28246 raft_consensus.cc:482] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:13:58.762689 28246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:13:58.763899 28246 raft_consensus.cc:504] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72a4a41209046e4bdf02cb7747a7326" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33689 } }
I20260620 12:13:58.764338 28246 leader_election.cc:302] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b72a4a41209046e4bdf02cb7747a7326; no voters:
I20260620 12:13:58.764706 28246 leader_election.cc:288] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:13:58.764910 28251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:13:58.765424 28251 raft_consensus.cc:686] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [term 1 LEADER]: Becoming Leader. State: Replica: b72a4a41209046e4bdf02cb7747a7326, State: Running, Role: LEADER
I20260620 12:13:58.765797 28251 consensus_queue.cc:237] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72a4a41209046e4bdf02cb7747a7326" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33689 } }
I20260620 12:13:58.766319 28246 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:13:58.767783 28252 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b72a4a41209046e4bdf02cb7747a7326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72a4a41209046e4bdf02cb7747a7326" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33689 } } }
I20260620 12:13:58.767947 28252 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:58.767904 28253 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b72a4a41209046e4bdf02cb7747a7326. Latest consensus state: current_term: 1 leader_uuid: "b72a4a41209046e4bdf02cb7747a7326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72a4a41209046e4bdf02cb7747a7326" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33689 } } }
I20260620 12:13:58.768127 28253 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326 [sys.catalog]: This master's current role is: LEADER
I20260620 12:13:58.768584 28262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:13:58.772459 28262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:13:58.779748 28262 catalog_manager.cc:1347] Generated new cluster ID: 254111b6e6b24d46acbec6d13b389ba6
I20260620 12:13:58.779850 28262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:13:58.812523 28262 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:13:58.814250 28262 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:13:58.821054 28262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b72a4a41209046e4bdf02cb7747a7326: Generated new TSK 0
I20260620 12:13:58.822269 28262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:13:58.862049 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33689
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:58.990291 28272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:58.990605 28272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:58.990666 28272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:58.995779 28272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:58.995894 28272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:58.995990 28272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:13:59.003221 28272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:59.004617 28272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:59.006063 28272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:59.017750 28278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:59.017794 28277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:59.018002 28272 server_base.cc:1034] running on GCE node
W20260620 12:13:59.017794 28280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:59.019208 28272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:59.020046 28272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:59.021241 28272 hybrid_clock.cc:648] HybridClock initialized: now 1781957639021198 us; error 58 us; skew 500 ppm
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:59.024538 28272 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:13:59.025744 28272 webserver.cc:460] Webserver started at http://127.24.141.1:43457/ using document root <none> and password file <none>
I20260620 12:13:59.026386 28272 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:59.026460 28272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:59.026659 28272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:59.028920 28272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "f3283a112ad248c683af6d264a2d15f0"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "04960eac6d4b09537d6158745109eaeb"
server_key_iv: "ef2f8cc0deff8ad674ceed389d0e270f"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.029601 28272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "f3283a112ad248c683af6d264a2d15f0"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "04960eac6d4b09537d6158745109eaeb"
server_key_iv: "ef2f8cc0deff8ad674ceed389d0e270f"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.034174 28272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:13:59.037360 28287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.039089 28272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:59.039341 28272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "f3283a112ad248c683af6d264a2d15f0"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "04960eac6d4b09537d6158745109eaeb"
server_key_iv: "ef2f8cc0deff8ad674ceed389d0e270f"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.039479 28272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:59.054653 28272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:59.058066 28272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:59.058246 28272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:59.058859 28272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:59.059788 28272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:59.059854 28272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.059898 28272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:59.059912 28272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.070204 28272 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:33859
I20260620 12:13:59.070245 28400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:33859 every 8 connection(s)
I20260620 12:13:59.071274 28272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:13:59.071671 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28272
I20260620 12:13:59.071800 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:13:59.072144 25140 external_mini_cluster.cc:1428] Setting key 2ebc248647612379574b725e7b23c0c1
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:59.092365 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:59.073524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:38051 (local address 127.24.141.62:33689)
0620 12:13:59.073837 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:59.073841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:59.075034 (+ 1193us) server_negotiation.cc:407] Connection header received
0620 12:13:59.076928 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:59.076931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:59.076993 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:59.077165 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:59.078880 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.079511 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.081425 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.081716 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.081835 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:59.087330 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:59.087334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:59.087336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:59.087374 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:59.089131 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.090007 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.090011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.090013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.090116 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.090619 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.090622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.090624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.090911 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:59.091033 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:59.091941 (+ 908us) server_negotiation.cc:299] Negotiation successful
0620 12:13:59.092132 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260620 12:13:59.093640 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:59.074197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33689 (local address 127.24.141.1:38051)
0620 12:13:59.074816 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:59.074878 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:59.076638 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:59.077390 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:59.077401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:59.077941 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:59.078680 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:59.078699 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.079673 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.079677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:59.081255 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:59.081270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.081923 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.081928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:59.082158 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:59.082914 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:59.082939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:59.087056 (+ 4117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:59.089351 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:59.089361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:59.089380 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:59.089841 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:59.090272 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:59.090275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:59.090277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:59.090490 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:59.091081 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:59.091089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:59.091621 (+ 532us) client_negotiation.cc:769] Sending connection context
0620 12:13:59.091850 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 12:13:59.092209 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":157,"threads_started":1}
I20260620 12:13:59.095584 28401 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33689
I20260620 12:13:59.095932 28401 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:59.096587 28401 heartbeater.cc:511] Master 127.24.141.62:33689 requested a full tablet report, sending...
I20260620 12:13:59.098505 28210 ts_manager.cc:194] Registered new tserver with Master: f3283a112ad248c683af6d264a2d15f0 (127.24.141.1:33859)
I20260620 12:13:59.099902 28210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:38051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:13:59.103916 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33689
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:59.234519 28406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:59.234874 28406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:59.234930 28406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:59.239346 28406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:59.239439 28406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:59.239504 28406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:13:59.245639 28406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:59.247080 28406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:59.248365 28406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:59.260015 28414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:59.260015 28412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:59.260353 28406 server_base.cc:1034] running on GCE node
W20260620 12:13:59.260061 28411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:59.261137 28406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:59.261976 28406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:59.263160 28406 hybrid_clock.cc:648] HybridClock initialized: now 1781957639263137 us; error 40 us; skew 500 ppm
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:59.266402 28406 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:13:59.267650 28406 webserver.cc:460] Webserver started at http://127.24.141.2:37353/ using document root <none> and password file <none>
I20260620 12:13:59.268491 28406 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:59.268594 28406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:59.268838 28406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:59.271459 28406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "879a033969784ce9a68c21b7b25df7e0"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "affd97a654b13a6115615d56ad87e89d"
server_key_iv: "4b1a88def9e281d8c34c1e94ab6be5c0"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.272277 28406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "879a033969784ce9a68c21b7b25df7e0"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "affd97a654b13a6115615d56ad87e89d"
server_key_iv: "4b1a88def9e281d8c34c1e94ab6be5c0"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.276932 28406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 12:13:59.279963 28421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.281888 28406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:13:59.282081 28406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "879a033969784ce9a68c21b7b25df7e0"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "affd97a654b13a6115615d56ad87e89d"
server_key_iv: "4b1a88def9e281d8c34c1e94ab6be5c0"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.282229 28406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:59.300334 28406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:59.304193 28406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:59.304395 28406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:59.305141 28406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:59.306126 28406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:59.306180 28406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.306239 28406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:59.306268 28406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.318011 28406 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:34239
I20260620 12:13:59.318042 28534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:34239 every 8 connection(s)
I20260620 12:13:59.319293 28406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:13:59.322639 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28406
I20260620 12:13:59.322790 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:13:59.323151 25140 external_mini_cluster.cc:1428] Setting key 85d7bd8c7e9b104b3f4b777c87adc2b7
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:13:59.338356 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:59.322017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:50377 (local address 127.24.141.62:33689)
0620 12:13:59.322220 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:59.322225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:59.323824 (+ 1599us) server_negotiation.cc:407] Connection header received
0620 12:13:59.325565 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:59.325569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:59.325641 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:59.325721 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:59.327160 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.327946 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.329674 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.330036 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.330194 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:59.333386 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:59.333392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:59.333396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:59.333447 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:59.335222 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.335995 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.335999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.336000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.336095 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.336639 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.336643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.336645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.336924 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:59.337042 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:59.338056 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0620 12:13:59.338212 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:13:59.339819 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:59.322660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33689 (local address 127.24.141.2:50377)
0620 12:13:59.323550 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:59.323659 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:59.325301 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:59.325961 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:59.325970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:59.326404 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:59.326977 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:59.326993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.328151 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.328155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:59.329466 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:59.329487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.330278 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.330282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:59.330525 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:59.331275 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:59.331297 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:59.333030 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:59.335422 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:59.335430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:59.335451 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:59.335846 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:59.336209 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:59.336212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:59.336213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:59.336483 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:59.337104 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:59.337113 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:59.337655 (+ 542us) client_negotiation.cc:769] Sending connection context
0620 12:13:59.337945 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 12:13:59.338466 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":171,"threads_started":1}
I20260620 12:13:59.341586 28535 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33689
I20260620 12:13:59.341997 28535 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:59.342808 28535 heartbeater.cc:511] Master 127.24.141.62:33689 requested a full tablet report, sending...
I20260620 12:13:59.344632 28210 ts_manager.cc:194] Registered new tserver with Master: 879a033969784ce9a68c21b7b25df7e0 (127.24.141.2:34239)
I20260620 12:13:59.345788 28210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:50377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:13:59.356046 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33689
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:13:59.494241 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:13:59.494585 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:13:59.494635 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:13:59.499138 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:13:59.499240 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:13:59.499329 28540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:13:59.505285 28540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:13:59.506662 28540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:13:59.507925 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:13:59.520068 28548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:59.520109 28546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:13:59.520123 28545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:13:59.520645 28540 server_base.cc:1034] running on GCE node
I20260620 12:13:59.521287 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:13:59.522184 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:13:59.523365 28540 hybrid_clock.cc:648] HybridClock initialized: now 1781957639523343 us; error 51 us; skew 500 ppm
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:13:59.526841 28540 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:13:59.528028 28540 webserver.cc:460] Webserver started at http://127.24.141.3:33359/ using document root <none> and password file <none>
I20260620 12:13:59.528734 28540 fs_manager.cc:359] Metadata directory not provided
I20260620 12:13:59.528810 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:13:59.529099 28540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:13:59.531895 28540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "ec9baccbf00149fc8e643a03a9b61d46"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "edd790107290a78b38cb5bb71e51462f"
server_key_iv: "c954b4cec0c9a9bfc53b092b379b4f02"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.532577 28540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "ec9baccbf00149fc8e643a03a9b61d46"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "edd790107290a78b38cb5bb71e51462f"
server_key_iv: "c954b4cec0c9a9bfc53b092b379b4f02"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.537662 28540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:13:59.540851 28555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.542950 28540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:13:59.543133 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "ec9baccbf00149fc8e643a03a9b61d46"
format_stamp: "Formatted at 2026-06-20 12:13:59 on dist-test-slave-rh5v"
server_key: "edd790107290a78b38cb5bb71e51462f"
server_key_iv: "c954b4cec0c9a9bfc53b092b379b4f02"
server_key_version: "encryptionkey@0"
I20260620 12:13:59.543298 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:13:59.564379 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:13:59.568797 28540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:13:59.569037 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:13:59.569845 28540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:13:59.570986 28540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:13:59.571036 28540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.571100 28540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:13:59.571128 28540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:13:59.582155 28540 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:38743
I20260620 12:13:59.582196 28668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:38743 every 8 connection(s)
I20260620 12:13:59.583321 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:13:59.585227 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28540
I20260620 12:13:59.585351 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:13:59.585656 25140 external_mini_cluster.cc:1428] Setting key c7fdba3a58ba8da112e1719d347b6c05
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:59.600935 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:59.585870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:55993 (local address 127.24.141.62:33689)
0620 12:13:59.586087 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:59.586093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:59.587470 (+ 1377us) server_negotiation.cc:407] Connection header received
0620 12:13:59.589190 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:59.589193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:59.589257 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:59.589326 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:59.590800 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.591345 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.592600 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.592979 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.593147 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:59.596348 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:59.596351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:59.596354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:59.596385 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:59.598013 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.598766 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.598773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.598776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.598979 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.599548 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.599553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.599554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.599855 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:59.599960 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:59.600588 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 12:13:59.600758 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:13:59.602021 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:59.586515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33689 (local address 127.24.141.3:55993)
0620 12:13:59.587239 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 12:13:59.587333 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 12:13:59.588919 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:13:59.589546 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:13:59.589555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:13:59.590031 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:13:59.590636 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:59.590652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.591475 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.591478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:59.592465 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:13:59.592477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.593239 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.593244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:13:59.593483 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:59.594196 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:13:59.594217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:13:59.596099 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:13:59.598183 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:59.598192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:59.598211 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:59.598575 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:59.599161 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:13:59.599165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:13:59.599167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:13:59.599428 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:13:59.599970 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:13:59.599979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:13:59.600323 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 12:13:59.600527 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 12:13:59.600923 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":143,"threads_started":1}
I20260620 12:13:59.603634 28669 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33689
I20260620 12:13:59.604070 28669 heartbeater.cc:464] Registering TS with master...
I20260620 12:13:59.604811 28669 heartbeater.cc:511] Master 127.24.141.62:33689 requested a full tablet report, sending...
I20260620 12:13:59.606528 28210 ts_manager.cc:194] Registered new tserver with Master: ec9baccbf00149fc8e643a03a9b61d46 (127.24.141.3:38743)
I20260620 12:13:59.607245 28210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:55993
I20260620 12:13:59.613233 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957639, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:13:59.643963 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:13:59.634016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.24.141.62:33689)
0620 12:13:59.634271 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 12:13:59.634277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:13:59.634444 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:13:59.634639 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:13:59.634642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:13:59.634688 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:13:59.634755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:13:59.635606 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.636137 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.636985 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:13:59.637262 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:13:59.637360 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:13:59.640271 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:13:59.640275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:13:59.640277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:13:59.640315 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:13:59.641847 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.642399 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.642402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.642404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.642510 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:13:59.642820 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:13:59.642824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:13:59.642825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:13:59.643104 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:13:59.643243 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:13:59.643659 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 12:13:59.643808 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260620 12:13:59.644574 28210 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52760
I20260620 12:13:59.645717 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28272
I20260620 12:13:59.651809 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28406
I20260620 12:13:59.659899 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28540
I20260620 12:13:59.666144 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28178
2026-06-20T12:13:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2263 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28681](info): set up 2 sockets
Jun 20 12:13:59 dist-test-slave-rh5v krb5kdc[28681](info): commencing operation
krb5kdc: starting...
W20260620 12:14:00.654174 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:00 dist-test-slave-rh5v krb5kdc[28681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:14:00.792553 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43523
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:00.938627 28694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:00.938988 28694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:00.939033 28694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:00.943403 28694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:00.943500 28694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:00.943521 28694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:00.943538 28694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:00.943554 28694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:00.949824 28694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:43523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:00.951364 28694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:00.952673 28694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:00.963238 28699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:00.963140 28700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:00.963219 28694 server_base.cc:1034] running on GCE node
W20260620 12:14:00.963140 28702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:00.964354 28694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:00.965888 28694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:00.967108 28694 hybrid_clock.cc:648] HybridClock initialized: now 1781957640967075 us; error 54 us; skew 500 ppm
Jun 20 12:14:00 dist-test-slave-rh5v krb5kdc[28681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957640, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:00.975178 28694 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:14:00.976400 28694 webserver.cc:460] Webserver started at http://127.24.141.62:37605/ using document root <none> and password file <none>
I20260620 12:14:00.977160 28694 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:00.977245 28694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:00.977484 28694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:00.979936 28694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "6a1441e9e62342c199d1d58d3b95f2bd"
format_stamp: "Formatted at 2026-06-20 12:14:00 on dist-test-slave-rh5v"
server_key: "fd2bf8d3ae276068fcd7bc1d68a237ca"
server_key_iv: "fca9741b69441d11884da2e92e1c8ef9"
server_key_version: "encryptionkey@0"
I20260620 12:14:00.980651 28694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "6a1441e9e62342c199d1d58d3b95f2bd"
format_stamp: "Formatted at 2026-06-20 12:14:00 on dist-test-slave-rh5v"
server_key: "fd2bf8d3ae276068fcd7bc1d68a237ca"
server_key_iv: "fca9741b69441d11884da2e92e1c8ef9"
server_key_version: "encryptionkey@0"
I20260620 12:14:00.985507 28694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:14:00.988828 28709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:00.991146 28694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:00.991353 28694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "6a1441e9e62342c199d1d58d3b95f2bd"
format_stamp: "Formatted at 2026-06-20 12:14:00 on dist-test-slave-rh5v"
server_key: "fd2bf8d3ae276068fcd7bc1d68a237ca"
server_key_iv: "fca9741b69441d11884da2e92e1c8ef9"
server_key_version: "encryptionkey@0"
I20260620 12:14:00.991498 28694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:01.017386 28694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:01.021890 28694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:01.022114 28694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:01.030846 28694 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:43523
I20260620 12:14:01.030895 28761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:43523 every 8 connection(s)
I20260620 12:14:01.032371 28694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:01.036854 28762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:01.041787 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28694
I20260620 12:14:01.041962 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:01.042303 25140 external_mini_cluster.cc:1428] Setting key d701d2f9840d4a42d6fd963742881de0
I20260620 12:14:01.045094 28762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd: Bootstrap starting.
I20260620 12:14:01.048473 28762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:01.049525 28762 log.cc:824] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:01.052377 28762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd: No bootstrap required, opened a new log
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:01.056469 28762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43523 } }
I20260620 12:14:01.056809 28762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:01.056849 28762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1441e9e62342c199d1d58d3b95f2bd, State: Initialized, Role: FOLLOWER
I20260620 12:14:01.057583 28762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43523 } }
I20260620 12:14:01.057772 28762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:01.057896 28762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:01.058015 28762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:01.059396 28762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43523 } }
I20260620 12:14:01.060418 28762 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1441e9e62342c199d1d58d3b95f2bd; no voters:
I20260620 12:14:01.060679 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.044288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.24.141.62:43523)
0620 12:14:01.044855 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:01.044874 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:01.044923 (+ 49us) server_negotiation.cc:407] Connection header received
0620 12:14:01.045871 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:01.045895 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:01.046335 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:01.046749 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:01.047759 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.048938 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.050222 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.050815 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.050995 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.054211 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:01.054218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:01.054239 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:01.054274 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:01.057154 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.057615 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.057623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.057630 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.057768 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.058059 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.058062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.058063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.058609 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:01.058838 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:01.059160 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 12:14:01.059543 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":154,"threads_started":1}
I20260620 12:14:01.060882 28762 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:01.061316 28767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:01.061533 28767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [term 1 LEADER]: Becoming Leader. State: Replica: 6a1441e9e62342c199d1d58d3b95f2bd, State: Running, Role: LEADER
I20260620 12:14:01.062028 28767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43523 } }
I20260620 12:14:01.063000 28762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:01.064504 28768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43523 } } }
I20260620 12:14:01.064706 28768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:01.064579 28769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1441e9e62342c199d1d58d3b95f2bd. Latest consensus state: current_term: 1 leader_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1441e9e62342c199d1d58d3b95f2bd" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43523 } } }
I20260620 12:14:01.064800 28769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:01.065233 28776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:01.069440 28776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:01.077873 28776 catalog_manager.cc:1347] Generated new cluster ID: f8ce0d644e5e43309890bb89991d8baa
I20260620 12:14:01.077993 28776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:01.103515 28776 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:01.104715 28776 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:01.114583 28776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1441e9e62342c199d1d58d3b95f2bd: Generated new TSK 0
I20260620 12:14:01.115473 28776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:14:01.152413 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43523
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:01.291114 28788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:01.291450 28788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:01.291517 28788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:01.296088 28788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:01.296186 28788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:01.296272 28788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:01.302320 28788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:01.303594 28788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:01.304778 28788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:01.316165 28796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:01.316185 28794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:01.316165 28793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:01.316609 28788 server_base.cc:1034] running on GCE node
I20260620 12:14:01.317319 28788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:01.318415 28788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:01.319674 28788 hybrid_clock.cc:648] HybridClock initialized: now 1781957641319638 us; error 47 us; skew 500 ppm
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:01.323375 28788 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:14:01.324738 28788 webserver.cc:460] Webserver started at http://127.24.141.1:37779/ using document root <none> and password file <none>
I20260620 12:14:01.325543 28788 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:01.325624 28788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:01.325851 28788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:01.328526 28788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "cf9a4eba12044d7b8efe755657831539"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "b1accb7fd0d1be7af388f91b8ded7524"
server_key_iv: "31fe362e0e7bb09a355bf0c9fdeb7a99"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.329270 28788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "cf9a4eba12044d7b8efe755657831539"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "b1accb7fd0d1be7af388f91b8ded7524"
server_key_iv: "31fe362e0e7bb09a355bf0c9fdeb7a99"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.334185 28788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:01.337646 28803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.339591 28788 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:01.339771 28788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "cf9a4eba12044d7b8efe755657831539"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "b1accb7fd0d1be7af388f91b8ded7524"
server_key_iv: "31fe362e0e7bb09a355bf0c9fdeb7a99"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.339934 28788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:01.356371 28788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:01.361377 28788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:01.361641 28788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:01.362458 28788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:01.363643 28788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:01.363699 28788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.363767 28788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:01.363801 28788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.375213 28788 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:40749
I20260620 12:14:01.375278 28916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:40749 every 8 connection(s)
I20260620 12:14:01.376368 28788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:01.382267 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28788
I20260620 12:14:01.382426 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:01.382732 25140 external_mini_cluster.cc:1428] Setting key 9b86e155fafb9450d9a2d331a7c75f0e
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:14:01.394788 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.378932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:54405 (local address 127.24.141.62:43523)
0620 12:14:01.379226 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:01.379230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:01.380289 (+ 1059us) server_negotiation.cc:407] Connection header received
0620 12:14:01.382024 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:01.382029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:01.382109 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:01.382259 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:01.383981 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.384707 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.386099 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.386423 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.386540 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.389639 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:01.389644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:01.389647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:01.389743 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:01.391502 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.392438 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.392444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.392445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.392600 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.393186 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.393189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.393191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.393469 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:01.393568 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:01.394356 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 12:14:01.394579 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":77}
I20260620 12:14:01.395799 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.379413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43523 (local address 127.24.141.1:54405)
0620 12:14:01.380070 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:01.380147 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:01.381662 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:01.382523 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:01.382532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:01.383026 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:01.383779 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:01.383803 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.384856 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.384859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:01.385939 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:01.385952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.386604 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.386608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:01.386801 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.387443 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:01.387462 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:01.389313 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:01.391664 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:01.391674 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:01.391702 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:01.392191 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:01.392751 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:01.392755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:01.392757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:01.393005 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:01.393598 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:01.393607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:01.394059 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 12:14:01.394263 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 12:14:01.394658 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:14:01.397431 28917 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43523
I20260620 12:14:01.397854 28917 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:01.398631 28917 heartbeater.cc:511] Master 127.24.141.62:43523 requested a full tablet report, sending...
I20260620 12:14:01.400892 28726 ts_manager.cc:194] Registered new tserver with Master: cf9a4eba12044d7b8efe755657831539 (127.24.141.1:40749)
I20260620 12:14:01.402792 28726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:54405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:14:01.415964 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43523
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:01.552845 28922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:01.553231 28922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:01.553285 28922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:01.557744 28922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:01.557821 28922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:01.557893 28922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:01.564071 28922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:01.565527 28922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:01.566807 28922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:01.578295 28930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:01.578349 28927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:01.578295 28928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:01.578621 28922 server_base.cc:1034] running on GCE node
I20260620 12:14:01.579159 28922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:01.579931 28922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:01.581169 28922 hybrid_clock.cc:648] HybridClock initialized: now 1781957641581130 us; error 52 us; skew 500 ppm
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:01.584260 28922 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:14:01.585405 28922 webserver.cc:460] Webserver started at http://127.24.141.2:42813/ using document root <none> and password file <none>
I20260620 12:14:01.586198 28922 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:01.586299 28922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:01.586545 28922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:01.589035 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "f6257fd70d0242e99f75335b410d9797"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "26356db3ad25eeb378586ac3f53ff69c"
server_key_iv: "e34d1dc4883f399dc6539177f8910b24"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.589792 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "f6257fd70d0242e99f75335b410d9797"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "26356db3ad25eeb378586ac3f53ff69c"
server_key_iv: "e34d1dc4883f399dc6539177f8910b24"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.594822 28922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:01.598369 28937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.600250 28922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:14:01.600468 28922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "f6257fd70d0242e99f75335b410d9797"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "26356db3ad25eeb378586ac3f53ff69c"
server_key_iv: "e34d1dc4883f399dc6539177f8910b24"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.600608 28922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:01.611785 28922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:01.615824 28922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:01.616070 28922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:01.616878 28922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:01.618002 28922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:01.618057 28922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.618108 28922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:01.618139 28922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.629112 28922 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:45881
I20260620 12:14:01.629137 29050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:45881 every 8 connection(s)
I20260620 12:14:01.630330 28922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:01.634943 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 28922
I20260620 12:14:01.635066 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:01.635368 25140 external_mini_cluster.cc:1428] Setting key 0c1f4799870fc499527240e9df15dcb6
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:14:01.650153 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.632961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:39239 (local address 127.24.141.62:43523)
0620 12:14:01.633153 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:01.633157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:01.634499 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 12:14:01.636218 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:01.636222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:01.636295 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:01.636377 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:01.638173 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.638752 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.640606 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.640989 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.641180 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.644886 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:01.644892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:01.644895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:01.644943 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:01.646711 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.647556 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.647559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.647561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.647658 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.648330 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.648333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.648334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.648621 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:01.648709 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:01.649778 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0620 12:14:01.649953 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:14:01.651403 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.633516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43523 (local address 127.24.141.2:39239)
0620 12:14:01.634259 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:01.634336 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:01.635939 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:01.636624 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:01.636633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:01.637205 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:01.637929 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:01.637953 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.638905 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.638908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:01.640383 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:01.640403 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.641284 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.641289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:01.641533 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.642508 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:01.642534 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:01.644595 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:01.646922 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:01.646930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:01.646949 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:01.647404 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:01.647817 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:01.647824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:01.647886 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:01.648187 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:01.648807 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:01.648818 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:01.649414 (+ 596us) client_negotiation.cc:769] Sending connection context
0620 12:14:01.649705 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 12:14:01.650131 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":175,"threads_started":1}
I20260620 12:14:01.653285 29051 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43523
I20260620 12:14:01.653630 29051 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:01.654268 29051 heartbeater.cc:511] Master 127.24.141.62:43523 requested a full tablet report, sending...
I20260620 12:14:01.655750 28726 ts_manager.cc:194] Registered new tserver with Master: f6257fd70d0242e99f75335b410d9797 (127.24.141.2:45881)
I20260620 12:14:01.656410 28726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:39239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:14:01.669615 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43523
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:01.800963 29056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:01.801340 29056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:01.801414 29056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:01.805951 29056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:01.806059 29056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:01.806154 29056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:01.812548 29056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:01.814169 29056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:01.815544 29056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:01.828049 29064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:01.828083 29056 server_base.cc:1034] running on GCE node
W20260620 12:14:01.828029 29062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:01.828029 29061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:01.828956 29056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:01.829834 29056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:01.831015 29056 hybrid_clock.cc:648] HybridClock initialized: now 1781957641830994 us; error 45 us; skew 500 ppm
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:01.834321 29056 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:14:01.835435 29056 webserver.cc:460] Webserver started at http://127.24.141.3:40201/ using document root <none> and password file <none>
I20260620 12:14:01.836122 29056 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:01.836189 29056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:01.836438 29056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:01.839171 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "ac6d720aa54445baa9b2c20633740c5f"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "d9e3d23670db53904a780f24b8bf8ba0"
server_key_iv: "d208a20a17ec7d342eea1cb64a0d6666"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.839810 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "ac6d720aa54445baa9b2c20633740c5f"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "d9e3d23670db53904a780f24b8bf8ba0"
server_key_iv: "d208a20a17ec7d342eea1cb64a0d6666"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.844949 29056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 12:14:01.848667 29071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.850772 29056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:01.850957 29056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "ac6d720aa54445baa9b2c20633740c5f"
format_stamp: "Formatted at 2026-06-20 12:14:01 on dist-test-slave-rh5v"
server_key: "d9e3d23670db53904a780f24b8bf8ba0"
server_key_iv: "d208a20a17ec7d342eea1cb64a0d6666"
server_key_version: "encryptionkey@0"
I20260620 12:14:01.851132 29056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:01.900880 29056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:01.905239 29056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:01.905442 29056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:01.906177 29056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:01.907181 29056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:01.907235 29056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.907286 29056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:01.907305 29056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:01.918406 29056 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:35029
I20260620 12:14:01.918488 29184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:35029 every 8 connection(s)
I20260620 12:14:01.919546 29056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:01.929834 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29056
I20260620 12:14:01.930008 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:01.930322 25140 external_mini_cluster.cc:1428] Setting key f3c9f81c5af179ba6052250e9295a18a
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:01.936434 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.922155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:34179 (local address 127.24.141.62:43523)
0620 12:14:01.922371 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:01.922375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:01.923514 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 12:14:01.925138 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:01.925142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:01.925209 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:01.925311 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:01.926863 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.927477 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.928866 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.929154 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.929259 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.932549 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:01.932553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:01.932555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:01.932586 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:01.934198 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.934782 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.934785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.934786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.934870 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.935222 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.935225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.935227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.935435 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:01.935515 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:01.936093 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 12:14:01.936250 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 12:14:01.937439 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.922621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43523 (local address 127.24.141.3:34179)
0620 12:14:01.923298 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:01.923367 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:01.924821 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:01.925552 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:01.925562 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:01.926008 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:01.926632 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:01.926650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.927654 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.927657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:01.928688 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:01.928703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.929375 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.929378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:01.929623 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.930420 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:01.930444 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:01.932335 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:01.934340 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:01.934347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:01.934370 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:01.934677 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:01.934970 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:01.934973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:01.934976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:01.935133 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:01.935543 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:01.935551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:01.935847 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 12:14:01.936036 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:14:01.936410 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":154,"threads_started":1}
I20260620 12:14:01.938776 29185 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43523
I20260620 12:14:01.939145 29185 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:01.939878 29185 heartbeater.cc:511] Master 127.24.141.62:43523 requested a full tablet report, sending...
I20260620 12:14:01.941480 28726 ts_manager.cc:194] Registered new tserver with Master: ac6d720aa54445baa9b2c20633740c5f (127.24.141.3:35029)
I20260620 12:14:01.942193 28726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:34179
I20260620 12:14:01.945631 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:14:01.977197 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.965246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.24.141.1:40749)
0620 12:14:01.965872 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:01.965880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:01.965915 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:14:01.965977 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:01.965983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:01.966198 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:01.966363 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:01.967241 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.968241 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.969306 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.969840 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.969958 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.972912 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:01.972930 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:01.972949 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:01.972997 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:01.974989 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.975521 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.975527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.975528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.975635 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.975871 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.975873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.975874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.976164 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:01.976359 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:01.976696 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 12:14:01.976922 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":242,"threads_started":1}
Jun 20 12:14:01 dist-test-slave-rh5v krb5kdc[28681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:01.990710 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:01.981181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.24.141.62:43523)
0620 12:14:01.981345 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:01.981351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:01.981438 (+ 87us) server_negotiation.cc:407] Connection header received
0620 12:14:01.981613 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:01.981616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:01.981662 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:01.981731 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:01.982476 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.983011 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.983835 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:01.984053 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:01.984146 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:01.987050 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:01.987054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:01.987057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:01.987102 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:01.988546 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.989211 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.989214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.989215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.989301 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:01.989558 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:01.989560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:01.989561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:01.989845 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:01.989971 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:01.990326 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 12:14:01.990507 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 12:14:01.991330 28726 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42876
I20260620 12:14:01.992198 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28788
I20260620 12:14:01.998898 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28922
I20260620 12:14:02.004897 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29056
I20260620 12:14:02.011739 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 28694
2026-06-20T12:14:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2345 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:02 dist-test-slave-rh5v krb5kdc[29197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:02 dist-test-slave-rh5v krb5kdc[29197](info): set up 2 sockets
Jun 20 12:14:02 dist-test-slave-rh5v krb5kdc[29197](info): commencing operation
krb5kdc: starting...
W20260620 12:14:02.985721 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:03 dist-test-slave-rh5v krb5kdc[29197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:14:03.122196 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33313
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:03.253182 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:03.253598 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:03.253669 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:03.258253 29210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:03.258342 29210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:03.258375 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:03.258399 29210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:03.258430 29210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:03.264429 29210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:03.265893 29210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:03.267216 29210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:03.277458 29216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:03.277477 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:03.277468 29215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:03.277737 29210 server_base.cc:1034] running on GCE node
I20260620 12:14:03.278595 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:03.280061 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:03.281247 29210 hybrid_clock.cc:648] HybridClock initialized: now 1781957643281213 us; error 64 us; skew 500 ppm
Jun 20 12:14:03 dist-test-slave-rh5v krb5kdc[29197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957643, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:03.284714 29210 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:14:03.285724 29210 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:03.285806 29210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:03.286017 29210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:03.288720 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "50ea575149da48548f16aaec939dddbb"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "ca31074f39d63ff4494a9d4f4ba38990"
server_key_iv: "0d8aa8fcf1d34da392475d16c1770fe2"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.289418 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "50ea575149da48548f16aaec939dddbb"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "ca31074f39d63ff4494a9d4f4ba38990"
server_key_iv: "0d8aa8fcf1d34da392475d16c1770fe2"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.294211 29210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:14:03.297384 29224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:03.299161 29210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:14:03.299381 29210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "50ea575149da48548f16aaec939dddbb"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "ca31074f39d63ff4494a9d4f4ba38990"
server_key_iv: "0d8aa8fcf1d34da392475d16c1770fe2"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.299525 29210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:03.311290 29210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:03.314909 29210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:03.315116 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:03.323439 29210 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:33313
I20260620 12:14:03.323812 29276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:33313 every 8 connection(s)
I20260620 12:14:03.324532 29210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:03.328595 29277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:03.331049 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29210
I20260620 12:14:03.331266 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:03.331573 25140 external_mini_cluster.cc:1428] Setting key e01b2d6513fc15de6360b7656189a3ba
I20260620 12:14:03.337878 29277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb: Bootstrap starting.
I20260620 12:14:03.341454 29277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:03.342478 29277 log.cc:824] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:14:03 dist-test-slave-rh5v krb5kdc[29197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:03.345224 29277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb: No bootstrap required, opened a new log
I20260620 12:14:03.349758 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:03.333563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.24.141.62:33313)
0620 12:14:03.334266 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:03.334293 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:03.334359 (+ 66us) server_negotiation.cc:407] Connection header received
0620 12:14:03.335427 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:03.335460 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:03.336069 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:03.336617 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:03.337598 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.339146 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.340251 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.341011 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.341262 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:03.344171 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:03.344180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:03.344204 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:03.344253 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:03.346356 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:03.346907 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:03.346915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:03.346921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:03.347059 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:03.347334 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:03.347337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:03.347339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:03.347845 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:03.348054 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:03.348360 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 12:14:03.348707 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":178,"threads_started":1}
I20260620 12:14:03.349680 29277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea575149da48548f16aaec939dddbb" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33313 } }
I20260620 12:14:03.350069 29277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:03.350148 29277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ea575149da48548f16aaec939dddbb, State: Initialized, Role: FOLLOWER
I20260620 12:14:03.350736 29277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea575149da48548f16aaec939dddbb" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33313 } }
I20260620 12:14:03.350906 29277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:03.350958 29277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:03.351071 29277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:03.352604 29277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea575149da48548f16aaec939dddbb" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33313 } }
I20260620 12:14:03.353165 29277 leader_election.cc:302] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ea575149da48548f16aaec939dddbb; no voters:
I20260620 12:14:03.353675 29277 leader_election.cc:288] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:03.353854 29282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:03.354085 29282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [term 1 LEADER]: Becoming Leader. State: Replica: 50ea575149da48548f16aaec939dddbb, State: Running, Role: LEADER
I20260620 12:14:03.354676 29282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea575149da48548f16aaec939dddbb" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33313 } }
I20260620 12:14:03.355265 29277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:03.356873 29284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ea575149da48548f16aaec939dddbb. Latest consensus state: current_term: 1 leader_uuid: "50ea575149da48548f16aaec939dddbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea575149da48548f16aaec939dddbb" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33313 } } }
I20260620 12:14:03.357105 29284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:03.357654 29292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:03.358469 29283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ea575149da48548f16aaec939dddbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea575149da48548f16aaec939dddbb" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33313 } } }
I20260620 12:14:03.358613 29283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:03.362020 29292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:03.370232 29292 catalog_manager.cc:1347] Generated new cluster ID: fed246b2c99f45508bf0a5f8ea872957
I20260620 12:14:03.370364 29292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:03.388290 29292 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:03.389524 29292 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:03.402272 29292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb: Generated new TSK 0
I20260620 12:14:03.403242 29292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:14:03.447705 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33313
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:03.600459 29303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:03.600816 29303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:03.600867 29303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:03.605175 29303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:03.605259 29303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:03.605326 29303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:03.611159 29303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:03.612491 29303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:03.613758 29303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:03.625677 29311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:03.625808 29309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:03.625670 29308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:03.626021 29303 server_base.cc:1034] running on GCE node
I20260620 12:14:03.626662 29303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:03.627511 29303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:03.628741 29303 hybrid_clock.cc:648] HybridClock initialized: now 1781957643628700 us; error 56 us; skew 500 ppm
Jun 20 12:14:03 dist-test-slave-rh5v krb5kdc[29197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957643, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:03.632197 29303 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:14:03.633208 29303 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:03.633303 29303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:03.633572 29303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:03.635970 29303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "92645114cacf886472e3ed9ecf06e8bf"
server_key_iv: "670ad9f0276faf35d19f9257b971c372"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.636600 29303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "92645114cacf886472e3ed9ecf06e8bf"
server_key_iv: "670ad9f0276faf35d19f9257b971c372"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.641583 29303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:14:03.644999 29317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:03.647009 29303 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:03.647215 29303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "92645114cacf886472e3ed9ecf06e8bf"
server_key_iv: "670ad9f0276faf35d19f9257b971c372"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.647353 29303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:03.678618 29303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:03.682718 29303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:03.682968 29303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:03.683710 29303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:03.684844 29303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:03.684908 29303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:03.684978 29303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:03.685010 29303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:03.696779 29303 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:35851
I20260620 12:14:03.696800 29430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:35851 every 8 connection(s)
I20260620 12:14:03.697569 29303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:03.699671 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29303
I20260620 12:14:03.699817 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:03.700186 25140 external_mini_cluster.cc:1428] Setting key b84e7b3ee0e5a24e58c9c7b4e52cc295
Jun 20 12:14:03 dist-test-slave-rh5v krb5kdc[29197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957643, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:14:03.717218 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:03.700123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:52347 (local address 127.24.141.62:33313)
0620 12:14:03.700329 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:03.700334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:03.702204 (+ 1870us) server_negotiation.cc:407] Connection header received
0620 12:14:03.703296 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:03.703299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:03.703360 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:03.703461 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:03.705264 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.705887 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.707724 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.708025 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.708140 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:03.712136 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:03.712143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:03.712147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:03.712200 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:03.713896 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:03.714842 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:03.714846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:03.714847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:03.714989 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:03.715509 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:03.715512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:03.715514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:03.715774 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:03.715871 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:03.716822 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 12:14:03.716988 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 12:14:03.718535 29433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:03.700694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33313 (local address 127.24.141.1:52347)
0620 12:14:03.701976 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:03.702064 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:03.703039 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:03.703702 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:03.703711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:03.704225 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:03.704995 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:03.705018 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.706063 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.706067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:03.707507 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:03.707526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.708381 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.708387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:03.708675 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:03.709593 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:03.709618 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:03.711751 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:03.714104 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:03.714113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:03.714136 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:03.714654 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:03.715147 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:03.715151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:03.715153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:03.715374 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:03.715927 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:03.715938 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:03.716430 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 12:14:03.716702 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 12:14:03.717256 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":263,"threads_started":1}
I20260620 12:14:03.720418 29431 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33313
I20260620 12:14:03.720886 29431 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:03.721740 29431 heartbeater.cc:511] Master 127.24.141.62:33313 requested a full tablet report, sending...
I20260620 12:14:03.724153 29237 ts_manager.cc:194] Registered new tserver with Master: 3d6980b67ee24d90a0c2e2fe6e0cc0d9 (127.24.141.1:35851)
I20260620 12:14:03.725925 29237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:52347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:14:03.736156 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33313
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:03.865624 29436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:03.865958 29436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:03.866007 29436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:03.871354 29436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:03.871474 29436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:03.871544 29436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:03.878120 29436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:03.879941 29436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:03.881269 29436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:03.893649 29444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:03.893669 29441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:03.893748 29442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:03.894515 29436 server_base.cc:1034] running on GCE node
I20260620 12:14:03.895058 29436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:03.895833 29436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:03.897048 29436 hybrid_clock.cc:648] HybridClock initialized: now 1781957643897016 us; error 51 us; skew 500 ppm
Jun 20 12:14:03 dist-test-slave-rh5v krb5kdc[29197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957643, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:03.900254 29436 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:14:03.901181 29436 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:03.901262 29436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:03.901479 29436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:03.903822 29436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "2e9bd45989df4a3ea163dadbe460cc47"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "b830058bc1daa6aba4d61fbacefeea2c"
server_key_iv: "61804bb074dec4cee1e61c74e9fc5f9c"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.904488 29436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "2e9bd45989df4a3ea163dadbe460cc47"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "b830058bc1daa6aba4d61fbacefeea2c"
server_key_iv: "61804bb074dec4cee1e61c74e9fc5f9c"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.909166 29436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 12:14:03.912479 29450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:03.914414 29436 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 12:14:03.914633 29436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "2e9bd45989df4a3ea163dadbe460cc47"
format_stamp: "Formatted at 2026-06-20 12:14:03 on dist-test-slave-rh5v"
server_key: "b830058bc1daa6aba4d61fbacefeea2c"
server_key_iv: "61804bb074dec4cee1e61c74e9fc5f9c"
server_key_version: "encryptionkey@0"
I20260620 12:14:03.914772 29436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:03.959512 29436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:03.963157 29436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:03.963380 29436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:03.964035 29436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:03.964984 29436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:03.965035 29436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:03.965128 29436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:03.965160 29436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:03.975368 29436 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:41503
I20260620 12:14:03.975466 29563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:41503 every 8 connection(s)
I20260620 12:14:03.976090 29436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:03.985790 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29436
I20260620 12:14:03.985919 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:03.986192 25140 external_mini_cluster.cc:1428] Setting key 921a2fa1ebf08c818efc3590e4d4c006
Jun 20 12:14:03 dist-test-slave-rh5v krb5kdc[29197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957643, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:03.993201 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:03.978574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:58025 (local address 127.24.141.62:33313)
0620 12:14:03.978769 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:03.978773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:03.980242 (+ 1469us) server_negotiation.cc:407] Connection header received
0620 12:14:03.981445 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:03.981456 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:03.981526 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:03.981609 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:03.983115 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.983738 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.985031 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.985351 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.985464 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:03.988943 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:03.988953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:03.988955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:03.988987 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:03.990635 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:03.991336 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:03.991339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:03.991340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:03.991427 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:03.991802 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:03.991806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:03.991808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:03.992022 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:03.992114 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:03.992708 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 12:14:03.992984 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 12:14:03.994270 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:03.979009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33313 (local address 127.24.141.2:58025)
0620 12:14:03.980027 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:03.980093 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:03.981195 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:03.981837 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:03.981847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:03.982319 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:03.982938 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:03.982964 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.983879 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.983883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:03.984891 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:03.984903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:03.985556 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:03.985561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:03.985750 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:03.986439 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:03.986459 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:03.988670 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:03.990837 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:03.990846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:03.990866 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:03.991216 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:03.991524 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:03.991527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:03.991529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:03.991705 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:03.992133 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:03.992141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:03.992451 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 12:14:03.992648 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 12:14:03.993216 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":138,"threads_started":1}
I20260620 12:14:03.996330 29564 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33313
I20260620 12:14:03.996686 29564 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:14:03.997309 29564 heartbeater.cc:511] Master 127.24.141.62:33313 requested a full tablet report, sending...
I20260620 12:14:03.998817 29237 ts_manager.cc:194] Registered new tserver with Master: 2e9bd45989df4a3ea163dadbe460cc47 (127.24.141.2:41503)
I20260620 12:14:03.999537 29237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:58025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:14:04.020462 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33313
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:04.156064 29569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:04.156524 29569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:04.156591 29569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:04.161578 29569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:04.161696 29569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:04.161772 29569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:04.168560 29569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:04.170325 29569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:04.171864 29569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:04.184207 29575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:04.184207 29574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:04.184211 29577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:04.184296 29569 server_base.cc:1034] running on GCE node
I20260620 12:14:04.185007 29569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:04.185886 29569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:04.187098 29569 hybrid_clock.cc:648] HybridClock initialized: now 1781957644187080 us; error 39 us; skew 500 ppm
Jun 20 12:14:04 dist-test-slave-rh5v krb5kdc[29197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957644, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:04.190760 29569 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:14:04.191697 29569 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:04.191795 29569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:04.192029 29569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:04.194571 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "10dd1944e1b04c5498913de1934767f5"
format_stamp: "Formatted at 2026-06-20 12:14:04 on dist-test-slave-rh5v"
server_key: "130aba94552e33268c1c37408b25de41"
server_key_iv: "55ef29f8df3adf0d74f71f44feb4cde7"
server_key_version: "encryptionkey@0"
I20260620 12:14:04.195235 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "10dd1944e1b04c5498913de1934767f5"
format_stamp: "Formatted at 2026-06-20 12:14:04 on dist-test-slave-rh5v"
server_key: "130aba94552e33268c1c37408b25de41"
server_key_iv: "55ef29f8df3adf0d74f71f44feb4cde7"
server_key_version: "encryptionkey@0"
I20260620 12:14:04.200016 29569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:14:04.203430 29583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:04.205508 29569 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:04.205695 29569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "10dd1944e1b04c5498913de1934767f5"
format_stamp: "Formatted at 2026-06-20 12:14:04 on dist-test-slave-rh5v"
server_key: "130aba94552e33268c1c37408b25de41"
server_key_iv: "55ef29f8df3adf0d74f71f44feb4cde7"
server_key_version: "encryptionkey@0"
I20260620 12:14:04.205842 29569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:04.233883 29569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:04.237775 29569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:04.238001 29569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:04.238683 29569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:04.239719 29569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:04.239794 29569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:04.239862 29569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:04.239887 29569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:04.250249 29569 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:34357
I20260620 12:14:04.250310 29696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:34357 every 8 connection(s)
I20260620 12:14:04.251259 29569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:04.259812 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29569
I20260620 12:14:04.259997 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:04.260370 25140 external_mini_cluster.cc:1428] Setting key 392090be7f04190ca6361d6aa10ff46b
Jun 20 12:14:04 dist-test-slave-rh5v krb5kdc[29197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957644, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:04.271098 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.253990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:40399 (local address 127.24.141.62:33313)
0620 12:14:04.254159 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.254163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.256108 (+ 1945us) server_negotiation.cc:407] Connection header received
0620 12:14:04.257437 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.257440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.257513 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.257698 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:04.259616 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.260478 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.262400 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.262770 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.262928 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.266369 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:04.266374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:04.266377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:04.266415 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:04.267903 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:04.268839 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:04.268842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:04.268844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:04.268963 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:04.269600 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:04.269604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:04.269605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:04.269881 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:04.269985 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.270745 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.270930 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:14:04.272162 29699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.254531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33313 (local address 127.24.141.3:40399)
0620 12:14:04.255828 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:04.255910 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:04.257117 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:04.257938 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:04.257948 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:04.258539 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:04.259275 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.259295 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.260716 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.260720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.262196 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.262213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.262997 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.263000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.263244 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.263971 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:04.263992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:04.266095 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:04.268169 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:04.268180 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:04.268201 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:04.268621 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:04.269208 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:04.269212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:04.269214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:04.269438 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:04.270046 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:04.270053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:04.270421 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 12:14:04.270637 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 12:14:04.271051 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":163,"threads_started":1}
I20260620 12:14:04.273970 29697 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33313
I20260620 12:14:04.274355 29697 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:04.275034 29697 heartbeater.cc:511] Master 127.24.141.62:33313 requested a full tablet report, sending...
I20260620 12:14:04.276422 29237 ts_manager.cc:194] Registered new tserver with Master: 10dd1944e1b04c5498913de1934767f5 (127.24.141.3:34357)
I20260620 12:14:04.277135 29237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:40399
I20260620 12:14:04.288319 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:14:04.299266 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.290930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.24.141.62:33313)
0620 12:14:04.291073 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.291077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.291253 (+ 176us) server_negotiation.cc:407] Connection header received
0620 12:14:04.291444 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.291447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.291505 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.291598 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:04.292384 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.292976 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.293978 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.294215 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.294322 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.295770 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:04.295773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:04.295775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:04.295801 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:04.297365 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:04.297874 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:04.297877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:04.297878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:04.297966 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:04.298253 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:04.298256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:04.298257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:04.298565 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:04.298684 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.298989 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.299111 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:14:04.303053 29237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:14:04.306594 29237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:04.326236 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.319269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.24.141.1:35851)
0620 12:14:04.320158 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.320166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.320194 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:14:04.320289 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.320295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.320550 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.321194 (+ 644us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:04.321331 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.322454 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.324387 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.325265 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.325702 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.325890 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.325952 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.326055 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":109,"threads_started":1}
I20260620 12:14:04.326256 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.318863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:35851 (local address 127.0.0.1:38256)
0620 12:14:04.319655 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:04.319723 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:04.319881 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:04.320753 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:04.320760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:04.320782 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:04.320962 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.320970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.322609 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.322613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.324212 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.324226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.325442 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.325446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.325637 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.325725 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 12:14:04.325837 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:14:04.326038 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":51,"thread_start_us":115,"threads_started":1}
I20260620 12:14:04.326263 29712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.320254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34357 (local address 127.0.0.1:47952)
0620 12:14:04.320603 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:04.320618 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:04.320716 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:04.321583 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:04.321586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:04.321606 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:04.321704 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.321710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.323394 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.323398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.324531 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.324539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.325596 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.325600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.325706 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.325728 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:14:04.325829 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 12:14:04.326037 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":38,"thread_start_us":101,"threads_started":1}
I20260620 12:14:04.326936 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.320591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.24.141.3:34357)
0620 12:14:04.321027 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.321040 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.321101 (+ 61us) server_negotiation.cc:407] Connection header received
0620 12:14:04.321178 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.321183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.321386 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.321562 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:04.321853 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.323241 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.324677 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.325472 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.326465 (+ 993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.326624 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.326676 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.326775 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":113,"threads_started":1}
I20260620 12:14:04.327035 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.318973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:41503 (local address 127.0.0.1:44216)
0620 12:14:04.320129 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:04.320142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:04.320226 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:04.320936 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:04.320939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:04.320965 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:04.321137 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.321145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.322591 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.322594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.324116 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.324128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.325551 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.325555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.325660 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.326760 (+ 1100us) client_negotiation.cc:769] Sending connection context
0620 12:14:04.326821 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 12:14:04.326910 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":128,"thread_start_us":187,"threads_started":1}
I20260620 12:14:04.327589 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.319682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.24.141.2:41503)
0620 12:14:04.320294 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.320308 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.320334 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:14:04.320424 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.320429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.320687 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.320861 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:04.321258 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.322431 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.324268 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.325191 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.325297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.325483 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.327265 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.327406 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":101,"threads_started":1}
I20260620 12:14:04.330435 29631 tablet_service.cc:1467] Processing CreateTablet for tablet b0550fcedb5d449eae5f0da759e82a3f (DEFAULT_TABLE table=test-table [id=2051957ed1ad45a9b2521ffe5326f2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:04.331140 29498 tablet_service.cc:1467] Processing CreateTablet for tablet b0550fcedb5d449eae5f0da759e82a3f (DEFAULT_TABLE table=test-table [id=2051957ed1ad45a9b2521ffe5326f2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:04.331763 29631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0550fcedb5d449eae5f0da759e82a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:04.332304 29498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0550fcedb5d449eae5f0da759e82a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:04.339135 29365 tablet_service.cc:1467] Processing CreateTablet for tablet b0550fcedb5d449eae5f0da759e82a3f (DEFAULT_TABLE table=test-table [id=2051957ed1ad45a9b2521ffe5326f2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:04.341293 29714 tablet_bootstrap.cc:492] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Bootstrap starting.
I20260620 12:14:04.342082 29365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0550fcedb5d449eae5f0da759e82a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:04.342545 29715 tablet_bootstrap.cc:492] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Bootstrap starting.
I20260620 12:14:04.344144 29714 tablet_bootstrap.cc:654] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:04.345007 29715 tablet_bootstrap.cc:654] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:04.345578 29714 log.cc:824] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:04.346169 29715 log.cc:824] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:04.348552 29715 tablet_bootstrap.cc:492] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: No bootstrap required, opened a new log
I20260620 12:14:04.348570 29714 tablet_bootstrap.cc:492] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: No bootstrap required, opened a new log
I20260620 12:14:04.348806 29715 ts_tablet_manager.cc:1397] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 12:14:04.348831 29714 ts_tablet_manager.cc:1397] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:14:04.351871 29718 tablet_bootstrap.cc:492] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9: Bootstrap starting.
I20260620 12:14:04.352517 29715 raft_consensus.cc:348] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.352881 29715 raft_consensus.cc:374] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:04.352938 29715 raft_consensus.cc:729] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e9bd45989df4a3ea163dadbe460cc47, State: Initialized, Role: FOLLOWER
I20260620 12:14:04.353372 29714 raft_consensus.cc:348] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.353571 29715 consensus_queue.cc:260] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.353672 29714 raft_consensus.cc:374] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:04.353735 29714 raft_consensus.cc:729] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dd1944e1b04c5498913de1934767f5, State: Initialized, Role: FOLLOWER
I20260620 12:14:04.354270 29714 consensus_queue.cc:260] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.354600 29564 heartbeater.cc:503] Master 127.24.141.62:33313 was elected leader, sending a full tablet report...
I20260620 12:14:04.354846 29715 ts_tablet_manager.cc:1428] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:14:04.355428 29714 ts_tablet_manager.cc:1428] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260620 12:14:04.355772 29697 heartbeater.cc:503] Master 127.24.141.62:33313 was elected leader, sending a full tablet report...
I20260620 12:14:04.356009 29718 tablet_bootstrap.cc:654] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:04.357016 29718 log.cc:824] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:04.359232 29718 tablet_bootstrap.cc:492] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9: No bootstrap required, opened a new log
I20260620 12:14:04.359469 29718 ts_tablet_manager.cc:1397] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:14:04.363801 29718 raft_consensus.cc:348] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.364179 29718 raft_consensus.cc:374] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:04.364252 29718 raft_consensus.cc:729] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6980b67ee24d90a0c2e2fe6e0cc0d9, State: Initialized, Role: FOLLOWER
I20260620 12:14:04.364832 29718 consensus_queue.cc:260] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.365720 29431 heartbeater.cc:503] Master 127.24.141.62:33313 was elected leader, sending a full tablet report...
I20260620 12:14:04.366094 29718 ts_tablet_manager.cc:1428] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260620 12:14:04.415882 29720 raft_consensus.cc:482] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:04.416185 29720 raft_consensus.cc:504] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.417791 29720 leader_election.cc:288] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d6980b67ee24d90a0c2e2fe6e0cc0d9 (127.24.141.1:35851), 2e9bd45989df4a3ea163dadbe460cc47 (127.24.141.2:41503)
I20260620 12:14:04.422055 29699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.418236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:41503 (local address 127.24.141.3:50999)
0620 12:14:04.418403 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:04.418423 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:04.418564 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:04.418894 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:04.418897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:04.418917 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:04.419069 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.419077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.419970 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.419974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.420874 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.420885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.421598 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.421602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.421721 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.421741 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:14:04.421809 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:14:04.421897 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":5,"spinlock_wait_cycles":46848}
I20260620 12:14:04.422053 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.418315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:50999 (local address 127.24.141.2:41503)
0620 12:14:04.418493 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.418498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.418520 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:14:04.418733 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.418736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.418798 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.418864 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:04.419200 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.419861 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.421011 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.421476 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.421598 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.421663 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.421835 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.421912 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:14:04.422451 29723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.418236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:35851 (local address 127.24.141.3:49669)
0620 12:14:04.418817 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:04.418830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:04.418998 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:04.419274 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:04.419277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:04.419328 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:04.419452 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.419459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.420276 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.420280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.421235 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:04.421245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.421949 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.421953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:04.422052 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.422076 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:14:04.422159 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 12:14:04.422255 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":164,"spinlock_wait_cycles":2304,"thread_start_us":170,"threads_started":1}
I20260620 12:14:04.422850 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.418393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:49669 (local address 127.24.141.1:35851)
0620 12:14:04.418529 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.418534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.418937 (+ 403us) server_negotiation.cc:407] Connection header received
0620 12:14:04.419109 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.419113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.419168 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.419250 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:04.419568 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.420141 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.421380 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.421826 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.421952 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.422015 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.422583 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.422692 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:14:04.422879 29518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0550fcedb5d449eae5f0da759e82a3f" candidate_uuid: "10dd1944e1b04c5498913de1934767f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e9bd45989df4a3ea163dadbe460cc47" is_pre_election: true
I20260620 12:14:04.423262 29518 raft_consensus.cc:2454] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10dd1944e1b04c5498913de1934767f5 in term 0.
I20260620 12:14:04.423540 29385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0550fcedb5d449eae5f0da759e82a3f" candidate_uuid: "10dd1944e1b04c5498913de1934767f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" is_pre_election: true
I20260620 12:14:04.423954 29385 raft_consensus.cc:2454] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10dd1944e1b04c5498913de1934767f5 in term 0.
I20260620 12:14:04.423921 29584 leader_election.cc:302] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10dd1944e1b04c5498913de1934767f5, 2e9bd45989df4a3ea163dadbe460cc47; no voters:
I20260620 12:14:04.424249 29720 raft_consensus.cc:2781] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:04.424367 29720 raft_consensus.cc:482] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:04.424413 29720 raft_consensus.cc:3037] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:04.425776 29720 raft_consensus.cc:504] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.426195 29720 leader_election.cc:288] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [CANDIDATE]: Term 1 election: Requested vote from peers 3d6980b67ee24d90a0c2e2fe6e0cc0d9 (127.24.141.1:35851), 2e9bd45989df4a3ea163dadbe460cc47 (127.24.141.2:41503)
I20260620 12:14:04.426775 29518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0550fcedb5d449eae5f0da759e82a3f" candidate_uuid: "10dd1944e1b04c5498913de1934767f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e9bd45989df4a3ea163dadbe460cc47"
I20260620 12:14:04.426743 29385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0550fcedb5d449eae5f0da759e82a3f" candidate_uuid: "10dd1944e1b04c5498913de1934767f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9"
I20260620 12:14:04.426925 29385 raft_consensus.cc:3037] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:04.426924 29518 raft_consensus.cc:3037] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:04.428368 29518 raft_consensus.cc:2454] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10dd1944e1b04c5498913de1934767f5 in term 1.
I20260620 12:14:04.428368 29385 raft_consensus.cc:2454] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10dd1944e1b04c5498913de1934767f5 in term 1.
I20260620 12:14:04.428802 29584 leader_election.cc:302] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10dd1944e1b04c5498913de1934767f5, 2e9bd45989df4a3ea163dadbe460cc47; no voters:
I20260620 12:14:04.429092 29720 raft_consensus.cc:2781] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:04.429430 29720 raft_consensus.cc:686] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 1 LEADER]: Becoming Leader. State: Replica: 10dd1944e1b04c5498913de1934767f5, State: Running, Role: LEADER
I20260620 12:14:04.429950 29720 consensus_queue.cc:237] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } }
I20260620 12:14:04.434499 29237 catalog_manager.cc:5495] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10dd1944e1b04c5498913de1934767f5 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "10dd1944e1b04c5498913de1934767f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10dd1944e1b04c5498913de1934767f5" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34357 } health_report { overall_health: HEALTHY } } }
W20260620 12:14:04.449141 29432 tablet.cc:2369] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:14:04.460880 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.456356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.24.141.3:34357)
0620 12:14:04.456499 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.456504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.456649 (+ 145us) server_negotiation.cc:407] Connection header received
0620 12:14:04.456834 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.456837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.456902 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.457031 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:04.457347 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.457895 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.459008 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:04.459262 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:04.459367 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:04.459879 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:04.460437 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:04.460556 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.460671 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.460745 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:14:04.469278 29385 raft_consensus.cc:1261] T b0550fcedb5d449eae5f0da759e82a3f P 3d6980b67ee24d90a0c2e2fe6e0cc0d9 [term 1 FOLLOWER]: Refusing update from remote peer 10dd1944e1b04c5498913de1934767f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:04.469308 29518 raft_consensus.cc:1261] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 1 FOLLOWER]: Refusing update from remote peer 10dd1944e1b04c5498913de1934767f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:04.470285 29724 consensus_queue.cc:1035] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6980b67ee24d90a0c2e2fe6e0cc0d9" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:04.470542 29720 consensus_queue.cc:1035] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e9bd45989df4a3ea163dadbe460cc47" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:14:04.477764 29565 tablet.cc:2369] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:14:04.479640 29728 mvcc.cc:205] Tried to move back new op lower bound from 7298898511735926784 to 7298898511588515840. Current Snapshot: MvccSnapshot[applied={T|T < 7298898511735926784}]
I20260620 12:14:04.480783 29727 mvcc.cc:205] Tried to move back new op lower bound from 7298898511735926784 to 7298898511588515840. Current Snapshot: MvccSnapshot[applied={T|T < 7298898511735926784 or (T in {7298898511735926784})}]
I20260620 12:14:04.490507 29241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:04.490777 29241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:04.495064 29241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50ea575149da48548f16aaec939dddbb: Sending DeleteTablet for 3 replicas of tablet b0550fcedb5d449eae5f0da759e82a3f
I20260620 12:14:04.496330 29365 tablet_service.cc:1514] Processing DeleteTablet for tablet b0550fcedb5d449eae5f0da759e82a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:04 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:38256
I20260620 12:14:04.496517 29498 tablet_service.cc:1514] Processing DeleteTablet for tablet b0550fcedb5d449eae5f0da759e82a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:04 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:44216
I20260620 12:14:04.496515 29631 tablet_service.cc:1514] Processing DeleteTablet for tablet b0550fcedb5d449eae5f0da759e82a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:04 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:47952
I20260620 12:14:04.497202 29739 tablet_replica.cc:330] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: stopping tablet replica
I20260620 12:14:04.497215 29738 tablet_replica.cc:330] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: stopping tablet replica
I20260620 12:14:04.497627 29739 raft_consensus.cc:2229] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:04.497637 29738 raft_consensus.cc:2229] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:04.497920 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29303
I20260620 12:14:04.498545 29738 raft_consensus.cc:2258] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:04.499297 29739 raft_consensus.cc:2258] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:04.501466 29739 ts_tablet_manager.cc:1905] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:04.501592 29738 ts_tablet_manager.cc:1905] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:04.504987 29738 ts_tablet_manager.cc:1918] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:04.505008 29739 ts_tablet_manager.cc:1918] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:04.505128 29738 log.cc:1196] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/b0550fcedb5d449eae5f0da759e82a3f
I20260620 12:14:04.505162 29739 log.cc:1196] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/b0550fcedb5d449eae5f0da759e82a3f
I20260620 12:14:04.505489 29738 ts_tablet_manager.cc:1939] T b0550fcedb5d449eae5f0da759e82a3f P 2e9bd45989df4a3ea163dadbe460cc47: Deleting consensus metadata
I20260620 12:14:04.505534 29739 ts_tablet_manager.cc:1939] T b0550fcedb5d449eae5f0da759e82a3f P 10dd1944e1b04c5498913de1934767f5: Deleting consensus metadata
I20260620 12:14:04.507146 29226 catalog_manager.cc:4841] TS 10dd1944e1b04c5498913de1934767f5 (127.24.141.3:34357): tablet b0550fcedb5d449eae5f0da759e82a3f (table test-table [id=2051957ed1ad45a9b2521ffe5326f2ef]) successfully deleted
I20260620 12:14:04.507158 29225 catalog_manager.cc:4841] TS 2e9bd45989df4a3ea163dadbe460cc47 (127.24.141.2:41503): tablet b0550fcedb5d449eae5f0da759e82a3f (table test-table [id=2051957ed1ad45a9b2521ffe5326f2ef]) successfully deleted
W20260620 12:14:04.509133 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:35851 (error 108)
I20260620 12:14:04.510020 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29436
I20260620 12:14:04.510541 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.510097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:35851 (local address 127.0.0.1:38266)
0620 12:14:04.510334 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:04.510422 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:35851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260620 12:14:04.512547 29226 catalog_manager.cc:4584] TS 3d6980b67ee24d90a0c2e2fe6e0cc0d9 (127.24.141.1:35851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0550fcedb5d449eae5f0da759e82a3f: Network error: Client connection negotiation failed: client connection to 127.24.141.1:35851: connect: Connection refused (error 111)
I20260620 12:14:04.519820 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29569
I20260620 12:14:04.527313 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29210
2026-06-20T12:14:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2517 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T12:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:04Z Disabled control of system clock
I20260620 12:14:04.563418 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33733
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:14:04.703061 29744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:04.703461 29744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:04.703516 29744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:04.708153 29744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:04.708273 29744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:04.708289 29744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:04.708307 29744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:04.708323 29744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:04.714105 29744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:33733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:04.715788 29744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:04.717727 29744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:04.728379 29749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:04.728390 29752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:04.728390 29750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:04.728842 29744 server_base.cc:1034] running on GCE node
I20260620 12:14:04.729768 29744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:04.731310 29744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:04.732534 29744 hybrid_clock.cc:648] HybridClock initialized: now 1781957644732475 us; error 58 us; skew 500 ppm
I20260620 12:14:04.735062 29744 webserver.cc:460] Webserver started at http://127.24.141.62:43589/ using document root <none> and password file <none>
I20260620 12:14:04.735852 29744 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:04.735934 29744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:04.736203 29744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:04.738950 29744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "f0c97df3715d42ebad9bec1c40cb8110"
format_stamp: "Formatted at 2026-06-20 12:14:04 on dist-test-slave-rh5v"
server_key: "ded44877753f782613332cd9105bb0c9"
server_key_iv: "c8782a6704fdaa33d5fcbb20558e4dfc"
server_key_version: "encryptionkey@0"
I20260620 12:14:04.739703 29744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "f0c97df3715d42ebad9bec1c40cb8110"
format_stamp: "Formatted at 2026-06-20 12:14:04 on dist-test-slave-rh5v"
server_key: "ded44877753f782613332cd9105bb0c9"
server_key_iv: "c8782a6704fdaa33d5fcbb20558e4dfc"
server_key_version: "encryptionkey@0"
I20260620 12:14:04.744611 29744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 12:14:04.748049 29758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:04.750007 29744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:04.750193 29744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "f0c97df3715d42ebad9bec1c40cb8110"
format_stamp: "Formatted at 2026-06-20 12:14:04 on dist-test-slave-rh5v"
server_key: "ded44877753f782613332cd9105bb0c9"
server_key_iv: "c8782a6704fdaa33d5fcbb20558e4dfc"
server_key_version: "encryptionkey@0"
I20260620 12:14:04.750344 29744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:04.756217 29744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:04.757027 29744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:04.757269 29744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:04.766419 29744 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:33733
I20260620 12:14:04.766474 29810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:33733 every 8 connection(s)
I20260620 12:14:04.768034 29744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:04.772650 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29744
I20260620 12:14:04.772811 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:04.772859 29811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:04.773175 25140 external_mini_cluster.cc:1428] Setting key f4fe625d5f15520c391906f33a719ae3
I20260620 12:14:04.780220 29811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110: Bootstrap starting.
I20260620 12:14:04.780709 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:04.775144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.24.141.62:33733)
0620 12:14:04.776049 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:04.776070 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:04.776117 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:14:04.776989 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:04.777013 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:04.777552 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:04.777963 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:04.778978 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:04.778984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:04.778999 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:04.779010 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:04.779082 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:04.779100 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:04.779239 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:04.779361 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 12:14:04.779694 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":200,"threads_started":1}
I20260620 12:14:04.783077 29811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:04.783993 29811 log.cc:824] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:04.786540 29811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110: No bootstrap required, opened a new log
I20260620 12:14:04.790489 29811 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c97df3715d42ebad9bec1c40cb8110" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33733 } }
I20260620 12:14:04.790802 29811 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:04.790839 29811 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c97df3715d42ebad9bec1c40cb8110, State: Initialized, Role: FOLLOWER
I20260620 12:14:04.791334 29811 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c97df3715d42ebad9bec1c40cb8110" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33733 } }
I20260620 12:14:04.791478 29811 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:04.791529 29811 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:04.791651 29811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:04.792713 29811 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c97df3715d42ebad9bec1c40cb8110" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33733 } }
I20260620 12:14:04.793149 29811 leader_election.cc:302] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0c97df3715d42ebad9bec1c40cb8110; no voters:
I20260620 12:14:04.793512 29811 leader_election.cc:288] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:04.793735 29816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:04.794126 29816 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [term 1 LEADER]: Becoming Leader. State: Replica: f0c97df3715d42ebad9bec1c40cb8110, State: Running, Role: LEADER
I20260620 12:14:04.794613 29816 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c97df3715d42ebad9bec1c40cb8110" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33733 } }
I20260620 12:14:04.795142 29811 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:04.796826 29817 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0c97df3715d42ebad9bec1c40cb8110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c97df3715d42ebad9bec1c40cb8110" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33733 } } }
I20260620 12:14:04.796876 29818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0c97df3715d42ebad9bec1c40cb8110. Latest consensus state: current_term: 1 leader_uuid: "f0c97df3715d42ebad9bec1c40cb8110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c97df3715d42ebad9bec1c40cb8110" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33733 } } }
I20260620 12:14:04.797025 29817 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:04.797129 29818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110 [sys.catalog]: This master's current role is: LEADER
W20260620 12:14:04.800597 29831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:14:04.800696 29831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:14:04.800798 29830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:04.801653 29830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:04.809564 29830 catalog_manager.cc:1347] Generated new cluster ID: 6cb9ba1d31a44a9d96c2372022e2c8c3
I20260620 12:14:04.809674 29830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:04.825016 29830 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:04.825474 29830 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:04.838256 29830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110: Generated new TSK 0
I20260620 12:14:04.839221 29830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:14:04.851147 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33733
--builtin_ntp_servers=127.24.141.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:14:04.982368 29835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:04.982708 29835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:04.982754 29835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:04.987125 29835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:04.987221 29835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:04.987285 29835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:04.993010 29835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:04.994729 29835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:04.996613 29835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:05.009788 29841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:05.009780 29840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:05.009780 29843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:05.010262 29835 server_base.cc:1034] running on GCE node
I20260620 12:14:05.010792 29835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:05.011893 29835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:05.013186 29835 hybrid_clock.cc:648] HybridClock initialized: now 1781957645013143 us; error 69 us; skew 500 ppm
I20260620 12:14:05.015775 29835 webserver.cc:460] Webserver started at http://127.24.141.1:45905/ using document root <none> and password file <none>
I20260620 12:14:05.016534 29835 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:05.016619 29835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:05.016865 29835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:05.019403 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "1b0e1d31f50b40d292379fc864bc96bc"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "97f0dc31681062fc6967f0d5bc4433e2"
server_key_iv: "d6502a1c2e265031c6821c09ad7592e6"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.020077 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "1b0e1d31f50b40d292379fc864bc96bc"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "97f0dc31681062fc6967f0d5bc4433e2"
server_key_iv: "d6502a1c2e265031c6821c09ad7592e6"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.025153 29835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 12:14:05.028828 29849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.030874 29835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:05.031080 29835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "1b0e1d31f50b40d292379fc864bc96bc"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "97f0dc31681062fc6967f0d5bc4433e2"
server_key_iv: "d6502a1c2e265031c6821c09ad7592e6"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.031237 29835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:05.036846 29835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:05.037806 29835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:05.038025 29835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:05.038681 29835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:05.039841 29835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:05.039896 29835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.039955 29835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:05.039985 29835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.050962 29835 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:45373
I20260620 12:14:05.051041 29962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:45373 every 8 connection(s)
I20260620 12:14:05.052065 29835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:05.059720 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29835
I20260620 12:14:05.059835 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:05.060119 25140 external_mini_cluster.cc:1428] Setting key bddaf61b423a48d6434ddaff966e19c8
I20260620 12:14:05.060360 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.054731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:43727 (local address 127.24.141.62:33733)
0620 12:14:05.054947 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.054952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.056430 (+ 1478us) server_negotiation.cc:407] Connection header received
0620 12:14:05.058224 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.058228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.058290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.058375 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.059760 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.059765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.059768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.059784 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.059812 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.059825 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.059933 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.060123 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.060187 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:14:05.061760 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.055239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33733 (local address 127.24.141.1:43727)
0620 12:14:05.056206 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.056268 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.057947 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.058634 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.058643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.058768 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.059438 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.059439 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.059494 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.059588 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.059975 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.059983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.059988 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.060174 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.060514 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":147,"threads_started":1}
I20260620 12:14:05.062672 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33733
--builtin_ntp_servers=127.24.141.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 12:14:05.063550 29963 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33733
I20260620 12:14:05.063921 29963 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:05.064482 29963 heartbeater.cc:511] Master 127.24.141.62:33733 requested a full tablet report, sending...
I20260620 12:14:05.066319 29775 ts_manager.cc:194] Registered new tserver with Master: 1b0e1d31f50b40d292379fc864bc96bc (127.24.141.1:45373)
W20260620 12:14:05.210845 29966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:05.211303 29966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:05.211372 29966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:05.216704 29966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:05.216816 29966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:05.216914 29966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:05.223170 29966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:05.224818 29966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:05.226756 29966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:05.241672 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:05.241647 29972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:05.241647 29974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:05.242017 29966 server_base.cc:1034] running on GCE node
I20260620 12:14:05.242664 29966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:05.243651 29966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:05.244877 29966 hybrid_clock.cc:648] HybridClock initialized: now 1781957645244853 us; error 51 us; skew 500 ppm
I20260620 12:14:05.247826 29966 webserver.cc:460] Webserver started at http://127.24.141.2:33479/ using document root <none> and password file <none>
I20260620 12:14:05.248696 29966 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:05.248786 29966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:05.249097 29966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:05.252140 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "047b63eed5e7438fb6708e5bdcb1fed4"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "3689a3075462ed00e1d3c41d0a3833af"
server_key_iv: "d515a972f0fba6719973b5c020624277"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.252946 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "047b63eed5e7438fb6708e5bdcb1fed4"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "3689a3075462ed00e1d3c41d0a3833af"
server_key_iv: "d515a972f0fba6719973b5c020624277"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.258682 29966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:05.262939 29980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.265195 29966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:14:05.265411 29966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "047b63eed5e7438fb6708e5bdcb1fed4"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "3689a3075462ed00e1d3c41d0a3833af"
server_key_iv: "d515a972f0fba6719973b5c020624277"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.265601 29966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:05.271811 29966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:05.272794 29966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:05.273003 29966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:05.273838 29966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:05.275089 29966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:05.275142 29966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.275204 29966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:05.275233 29966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.287317 29966 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:44951
I20260620 12:14:05.287385 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:44951 every 8 connection(s)
I20260620 12:14:05.288614 29966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:05.294099 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 29966
I20260620 12:14:05.294286 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:05.294680 25140 external_mini_cluster.cc:1428] Setting key 1ca3892d7e48c72acbf9ee3720121985
I20260620 12:14:05.297442 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33733
--builtin_ntp_servers=127.24.141.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 12:14:05.297739 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.291206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:56865 (local address 127.24.141.62:33733)
0620 12:14:05.291398 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.291402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.292917 (+ 1515us) server_negotiation.cc:407] Connection header received
0620 12:14:05.295110 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.295116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.295214 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.295326 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.297120 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.297125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.297127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.297140 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.297165 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.297173 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.297256 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.297447 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.297509 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:14:05.299281 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.291614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33733 (local address 127.24.141.2:56865)
0620 12:14:05.292642 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.292706 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.294645 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.295625 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.295637 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.295796 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.296728 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.296730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.296797 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.296915 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.297254 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.297264 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.297271 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.298022 (+ 751us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.298326 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":160,"threads_started":1}
I20260620 12:14:05.301026 30094 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33733
I20260620 12:14:05.301395 30094 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:05.302042 30094 heartbeater.cc:511] Master 127.24.141.62:33733 requested a full tablet report, sending...
I20260620 12:14:05.304065 29775 ts_manager.cc:194] Registered new tserver with Master: 047b63eed5e7438fb6708e5bdcb1fed4 (127.24.141.2:44951)
W20260620 12:14:05.443687 30097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:05.444049 30097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:05.444113 30097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:05.449095 30097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:05.449189 30097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:05.449277 30097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:05.455950 30097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:05.457584 30097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:05.459713 30097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:05.475461 30105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:05.475471 30102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:05.475497 30103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:05.476060 30097 server_base.cc:1034] running on GCE node
I20260620 12:14:05.476701 30097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:05.477696 30097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:05.479036 30097 hybrid_clock.cc:648] HybridClock initialized: now 1781957645478993 us; error 81 us; skew 500 ppm
I20260620 12:14:05.481705 30097 webserver.cc:460] Webserver started at http://127.24.141.3:44219/ using document root <none> and password file <none>
I20260620 12:14:05.482643 30097 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:05.482758 30097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:05.483057 30097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:05.486601 30097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "c47a0d9df7c84736bac25c6ddbb2cd67"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "3aa1022094b8658dd85677cc0580b85c"
server_key_iv: "b67cf3030f699601d2379764e4df2836"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.487625 30097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "c47a0d9df7c84736bac25c6ddbb2cd67"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "3aa1022094b8658dd85677cc0580b85c"
server_key_iv: "b67cf3030f699601d2379764e4df2836"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.494062 30097 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 12:14:05.498167 30111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.500005 30097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:05.500164 30097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "c47a0d9df7c84736bac25c6ddbb2cd67"
format_stamp: "Formatted at 2026-06-20 12:14:05 on dist-test-slave-rh5v"
server_key: "3aa1022094b8658dd85677cc0580b85c"
server_key_iv: "b67cf3030f699601d2379764e4df2836"
server_key_version: "encryptionkey@0"
I20260620 12:14:05.500324 30097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:05.506114 30097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:05.506928 30097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:05.507114 30097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:05.507812 30097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:05.509123 30097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:05.509194 30097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.509258 30097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:05.509282 30097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:05.521121 30097 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:41501
I20260620 12:14:05.521162 30224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:41501 every 8 connection(s)
I20260620 12:14:05.522480 30097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:05.527873 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30097
I20260620 12:14:05.528021 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:05.528383 25140 external_mini_cluster.cc:1428] Setting key 108b280abe924fa7f27c5de62faa9276
I20260620 12:14:05.532792 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.525455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:36643 (local address 127.24.141.62:33733)
0620 12:14:05.525699 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.525705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.527668 (+ 1963us) server_negotiation.cc:407] Connection header received
0620 12:14:05.530150 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.530153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.530222 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.530313 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.532220 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.532224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.532227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.532240 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.532269 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.532278 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.532375 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.532591 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.532648 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 12:14:05.533890 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.526178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33733 (local address 127.24.141.3:36643)
0620 12:14:05.527411 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.527494 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.529691 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.530697 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.530711 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.530885 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.531752 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.531755 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.531860 (+ 105us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.531990 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.532374 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.532380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.532385 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.532568 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.532847 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":159,"threads_started":1}
I20260620 12:14:05.535549 30225 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33733
I20260620 12:14:05.535874 30225 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:05.536448 30225 heartbeater.cc:511] Master 127.24.141.62:33733 requested a full tablet report, sending...
I20260620 12:14:05.537885 29775 ts_manager.cc:194] Registered new tserver with Master: c47a0d9df7c84736bac25c6ddbb2cd67 (127.24.141.3:41501)
I20260620 12:14:05.544119 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:14:05.548380 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.546502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.24.141.62:33733)
0620 12:14:05.546694 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.546699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.546790 (+ 91us) server_negotiation.cc:407] Connection header received
0620 12:14:05.546976 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.546980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.547024 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.547094 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.547922 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.547925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.547928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.547938 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.547960 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.547969 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.548063 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.548149 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.548201 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 12:14:05.552305 29775 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:14:05.556464 29775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:05.574033 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.570554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.24.141.1:45373)
0620 12:14:05.570872 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.570880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.571618 (+ 738us) server_negotiation.cc:407] Connection header received
0620 12:14:05.571723 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.571729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.572009 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.572172 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.573357 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.573364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.573376 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.573392 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.573436 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.573469 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.573597 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.573770 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.573858 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260620 12:14:05.574007 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.569573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45373 (local address 127.0.0.1:43774)
0620 12:14:05.571471 (+ 1898us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.571484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.571611 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.572158 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.572168 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.572278 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.573016 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.573018 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.573044 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.573166 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.573585 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.573597 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.573604 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.573731 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.573830 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1831,"thread_start_us":157,"threads_started":1}
I20260620 12:14:05.574249 30240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.570707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39778 (local address 127.24.141.2:44951)
0620 12:14:05.571333 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.571347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.571965 (+ 618us) server_negotiation.cc:407] Connection header received
0620 12:14:05.572046 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.572052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.572266 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.572419 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.573500 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.573508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.573521 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.573536 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.573591 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.573636 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.573734 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.573928 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.574041 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":118,"threads_started":1}
I20260620 12:14:05.574796 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.570881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.24.141.3:41501)
0620 12:14:05.571402 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.571422 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.571556 (+ 134us) server_negotiation.cc:407] Connection header received
0620 12:14:05.571655 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.571662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.571933 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.573284 (+ 1351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.573968 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.573973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.573983 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.573993 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.574031 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.574054 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.574519 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.574568 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.574639 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260620 12:14:05.574394 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.569553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:41501 (local address 127.0.0.1:50420)
0620 12:14:05.570097 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.570166 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.570311 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.572156 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.572164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.572281 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.573250 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.573252 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.573259 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.573274 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.574162 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.574165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.574169 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.574223 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.574267 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":177,"threads_started":1}
I20260620 12:14:05.574647 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.569648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:44951 (local address 127.0.0.1:39778)
0620 12:14:05.571827 (+ 2179us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.571839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.571930 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.572667 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.572670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.572685 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.573177 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.573178 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.573193 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.573211 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.573807 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.573811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.573813 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.574501 (+ 688us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.574538 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2119,"thread_start_us":139,"threads_started":1}
I20260620 12:14:05.578858 30028 tablet_service.cc:1467] Processing CreateTablet for tablet cf682f74feb64d87877d0be2564b7efd (DEFAULT_TABLE table=test-table [id=7d6e12383f6d4a399f159ec2e1e35f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:05.578858 30159 tablet_service.cc:1467] Processing CreateTablet for tablet cf682f74feb64d87877d0be2564b7efd (DEFAULT_TABLE table=test-table [id=7d6e12383f6d4a399f159ec2e1e35f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:05.578862 29897 tablet_service.cc:1467] Processing CreateTablet for tablet cf682f74feb64d87877d0be2564b7efd (DEFAULT_TABLE table=test-table [id=7d6e12383f6d4a399f159ec2e1e35f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:05.580340 29897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf682f74feb64d87877d0be2564b7efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:05.580343 30028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf682f74feb64d87877d0be2564b7efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:05.580343 30159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf682f74feb64d87877d0be2564b7efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:05.589576 30242 tablet_bootstrap.cc:492] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc: Bootstrap starting.
I20260620 12:14:05.591094 30243 tablet_bootstrap.cc:492] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: Bootstrap starting.
I20260620 12:14:05.591094 30244 tablet_bootstrap.cc:492] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: Bootstrap starting.
I20260620 12:14:05.592538 30242 tablet_bootstrap.cc:654] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:05.593601 30242 log.cc:824] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:05.594267 30244 tablet_bootstrap.cc:654] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:05.594267 30243 tablet_bootstrap.cc:654] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:05.595755 30243 log.cc:824] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:05.596163 30244 log.cc:824] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:05.596695 30242 tablet_bootstrap.cc:492] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc: No bootstrap required, opened a new log
I20260620 12:14:05.596998 30242 ts_tablet_manager.cc:1397] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:14:05.598482 30243 tablet_bootstrap.cc:492] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: No bootstrap required, opened a new log
I20260620 12:14:05.598482 30244 tablet_bootstrap.cc:492] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: No bootstrap required, opened a new log
I20260620 12:14:05.598747 30243 ts_tablet_manager.cc:1397] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:14:05.598747 30244 ts_tablet_manager.cc:1397] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 12:14:05.601974 30242 raft_consensus.cc:348] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.602411 30242 raft_consensus.cc:374] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:05.602478 30242 raft_consensus.cc:729] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0e1d31f50b40d292379fc864bc96bc, State: Initialized, Role: FOLLOWER
I20260620 12:14:05.603116 30242 consensus_queue.cc:260] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.604032 29963 heartbeater.cc:503] Master 127.24.141.62:33733 was elected leader, sending a full tablet report...
I20260620 12:14:05.603794 30243 raft_consensus.cc:348] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.603793 30244 raft_consensus.cc:348] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.604204 30244 raft_consensus.cc:374] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:05.604204 30243 raft_consensus.cc:374] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:05.604264 30244 raft_consensus.cc:729] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047b63eed5e7438fb6708e5bdcb1fed4, State: Initialized, Role: FOLLOWER
I20260620 12:14:05.604264 30243 raft_consensus.cc:729] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47a0d9df7c84736bac25c6ddbb2cd67, State: Initialized, Role: FOLLOWER
I20260620 12:14:05.604602 30242 ts_tablet_manager.cc:1428] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 12:14:05.604885 30243 consensus_queue.cc:260] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.604885 30244 consensus_queue.cc:260] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.606243 30225 heartbeater.cc:503] Master 127.24.141.62:33733 was elected leader, sending a full tablet report...
I20260620 12:14:05.606274 30094 heartbeater.cc:503] Master 127.24.141.62:33733 was elected leader, sending a full tablet report...
I20260620 12:14:05.606395 30244 ts_tablet_manager.cc:1428] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:14:05.608300 30243 ts_tablet_manager.cc:1428] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
W20260620 12:14:05.773923 30226 tablet.cc:2369] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:14:05.790135 30095 tablet.cc:2369] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:14:05.803700 29964 tablet.cc:2369] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:14:05.838857 30250 raft_consensus.cc:482] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:05.839195 30250 raft_consensus.cc:504] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.841009 30250 leader_election.cc:288] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b0e1d31f50b40d292379fc864bc96bc (127.24.141.1:45373), 047b63eed5e7438fb6708e5bdcb1fed4 (127.24.141.2:44951)
I20260620 12:14:05.843850 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.841527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:44951 (local address 127.24.141.3:44663)
0620 12:14:05.841810 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.841829 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.841971 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.842266 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.842269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.842285 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.843087 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.843091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.843112 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.843180 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.843523 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.843527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.843530 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.843622 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.843675 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260620 12:14:05.843914 30240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.841630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:44663 (local address 127.24.141.2:44951)
0620 12:14:05.841802 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.841807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.841949 (+ 142us) server_negotiation.cc:407] Connection header received
0620 12:14:05.842091 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.842095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.842149 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.842233 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.843372 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.843376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.843378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.843390 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.843412 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.843420 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.843523 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.843681 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.843741 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:14:05.843932 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.841860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:43335 (local address 127.24.141.1:45373)
0620 12:14:05.842019 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.842025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.842160 (+ 135us) server_negotiation.cc:407] Connection header received
0620 12:14:05.842228 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.842231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.842277 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.842376 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.843372 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.843377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.843380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.843399 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.843426 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.843436 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.843556 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.843741 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.843792 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:14:05.844024 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.841527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45373 (local address 127.24.141.3:43335)
0620 12:14:05.842010 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:05.842024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:05.842148 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:05.842467 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:05.842469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:05.842482 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:05.843151 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:05.843152 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:05.843163 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:05.843190 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:05.843559 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:05.843562 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:05.843615 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 12:14:05.843735 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 12:14:05.843771 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":31,"thread_start_us":181,"threads_started":1}
I20260620 12:14:05.844971 29917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf682f74feb64d87877d0be2564b7efd" candidate_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0e1d31f50b40d292379fc864bc96bc" is_pre_election: true
I20260620 12:14:05.845237 30048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf682f74feb64d87877d0be2564b7efd" candidate_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" is_pre_election: true
I20260620 12:14:05.845472 29917 raft_consensus.cc:2454] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c47a0d9df7c84736bac25c6ddbb2cd67 in term 0.
I20260620 12:14:05.845733 30048 raft_consensus.cc:2454] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c47a0d9df7c84736bac25c6ddbb2cd67 in term 0.
I20260620 12:14:05.846166 30113 leader_election.cc:302] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0e1d31f50b40d292379fc864bc96bc, c47a0d9df7c84736bac25c6ddbb2cd67; no voters:
I20260620 12:14:05.846499 30250 raft_consensus.cc:2781] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:05.846604 30250 raft_consensus.cc:482] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:05.846647 30250 raft_consensus.cc:3037] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:05.907802 30250 raft_consensus.cc:504] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.908483 30250 leader_election.cc:288] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [CANDIDATE]: Term 1 election: Requested vote from peers 1b0e1d31f50b40d292379fc864bc96bc (127.24.141.1:45373), 047b63eed5e7438fb6708e5bdcb1fed4 (127.24.141.2:44951)
I20260620 12:14:05.909211 30048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf682f74feb64d87877d0be2564b7efd" candidate_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047b63eed5e7438fb6708e5bdcb1fed4"
I20260620 12:14:05.909194 29917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf682f74feb64d87877d0be2564b7efd" candidate_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0e1d31f50b40d292379fc864bc96bc"
I20260620 12:14:05.909385 30048 raft_consensus.cc:3037] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:05.909417 29917 raft_consensus.cc:3037] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:05.910766 30048 raft_consensus.cc:2454] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c47a0d9df7c84736bac25c6ddbb2cd67 in term 1.
I20260620 12:14:05.910976 29917 raft_consensus.cc:2454] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c47a0d9df7c84736bac25c6ddbb2cd67 in term 1.
I20260620 12:14:05.911288 30112 leader_election.cc:302] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047b63eed5e7438fb6708e5bdcb1fed4, c47a0d9df7c84736bac25c6ddbb2cd67; no voters:
I20260620 12:14:05.911505 30250 raft_consensus.cc:2781] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:05.911830 30250 raft_consensus.cc:686] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 1 LEADER]: Becoming Leader. State: Replica: c47a0d9df7c84736bac25c6ddbb2cd67, State: Running, Role: LEADER
I20260620 12:14:05.912343 30250 consensus_queue.cc:237] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } }
I20260620 12:14:05.916496 29774 catalog_manager.cc:5495] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 reported cstate change: term changed from 0 to 1, leader changed from <none> to c47a0d9df7c84736bac25c6ddbb2cd67 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47a0d9df7c84736bac25c6ddbb2cd67" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 } health_report { overall_health: UNKNOWN } } }
I20260620 12:14:05.951514 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:05.949682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.24.141.3:41501)
0620 12:14:05.949877 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:05.949882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:05.949905 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:05.950023 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:05.950026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:05.950108 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:05.950190 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:05.951047 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:05.951051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:05.951054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:05.951065 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:05.951085 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:05.951093 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:05.951207 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:05.951249 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 12:14:05.951298 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 12:14:05.960044 29917 raft_consensus.cc:1261] T cf682f74feb64d87877d0be2564b7efd P 1b0e1d31f50b40d292379fc864bc96bc [term 1 FOLLOWER]: Refusing update from remote peer c47a0d9df7c84736bac25c6ddbb2cd67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:05.960170 30048 raft_consensus.cc:1261] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 1 FOLLOWER]: Refusing update from remote peer c47a0d9df7c84736bac25c6ddbb2cd67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:05.961134 30250 consensus_queue.cc:1035] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047b63eed5e7438fb6708e5bdcb1fed4" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:05.961422 30252 consensus_queue.cc:1035] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0e1d31f50b40d292379fc864bc96bc" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:05.972446 30256 mvcc.cc:205] Tried to move back new op lower bound from 7298898517841354752 to 7298898517660381184. Current Snapshot: MvccSnapshot[applied={T|T < 7298898517841354752}]
I20260620 12:14:05.984712 29775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:05.985111 29775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:05.988552 29775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0c97df3715d42ebad9bec1c40cb8110: Sending DeleteTablet for 3 replicas of tablet cf682f74feb64d87877d0be2564b7efd
I20260620 12:14:05.989957 30159 tablet_service.cc:1514] Processing DeleteTablet for tablet cf682f74feb64d87877d0be2564b7efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:05 UTC) from {username='slave'} at 127.0.0.1:50420
I20260620 12:14:05.989964 30028 tablet_service.cc:1514] Processing DeleteTablet for tablet cf682f74feb64d87877d0be2564b7efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:05 UTC) from {username='slave'} at 127.0.0.1:39778
I20260620 12:14:05.990708 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29835
I20260620 12:14:05.990705 30265 tablet_replica.cc:330] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: stopping tablet replica
I20260620 12:14:05.990718 29897 tablet_service.cc:1514] Processing DeleteTablet for tablet cf682f74feb64d87877d0be2564b7efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:05 UTC) from {username='slave'} at 127.0.0.1:43774
I20260620 12:14:05.991127 30265 raft_consensus.cc:2229] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:05.991472 30266 tablet_replica.cc:330] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: stopping tablet replica
I20260620 12:14:05.991875 30266 raft_consensus.cc:2229] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:05.992357 30266 raft_consensus.cc:2258] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:05.994349 30265 raft_consensus.cc:2258] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:05.994601 30266 ts_tablet_manager.cc:1905] T cf682f74feb64d87877d0be2564b7efd P 047b63eed5e7438fb6708e5bdcb1fed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:05.996492 30265 ts_tablet_manager.cc:1905] T cf682f74feb64d87877d0be2564b7efd P c47a0d9df7c84736bac25c6ddbb2cd67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:06.000604 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29966
W20260620 12:14:06.000619 29760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:45373 (error 108)
I20260620 12:14:06.003577 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:06.003122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45373 (local address 127.0.0.1:43784)
0620 12:14:06.003357 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:06.003438 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260620 12:14:06.004007 29760 catalog_manager.cc:4584] TS 1b0e1d31f50b40d292379fc864bc96bc (127.24.141.1:45373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf682f74feb64d87877d0be2564b7efd: Network error: Client connection negotiation failed: client connection to 127.24.141.1:45373: connect: Connection refused (error 111)
I20260620 12:14:06.008968 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:06.008663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:44951 (local address 127.0.0.1:39782)
0620 12:14:06.008801 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:06.008855 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.2:44951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 12:14:06.009212 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30097
I20260620 12:14:06.016690 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 29744
2026-06-20T12:14:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1488 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 12:14:06.041905 25140 mini_oidc.cc:152] Starting JWKS server
I20260620 12:14:06.043975 25140 webserver.cc:460] Webserver started at https://127.0.0.1:46537/ using document root <none> and password file <none>
I20260620 12:14:06.044193 25140 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:14:06.044560 25140 webserver.cc:460] Webserver started at http://127.0.0.1:41541/ using document root <none> and password file <none>
2026-06-20T12:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:06Z Disabled control of system clock
I20260620 12:14:06.056959 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:36931
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:36931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46537/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/testchainca.cert with env {}
W20260620 12:14:06.198660 30273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:06.199210 30273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:06.199359 30273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:06.204695 30273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:06.204818 30273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:06.204835 30273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:06.204854 30273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:06.204871 30273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:14:06.204888 30273 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:14:06.204900 30273 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46537/jwks.json
I20260620 12:14:06.211796 30273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:36931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46537/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:06.213795 30273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:06.215359 30273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:06.227604 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:06.227681 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:06.227604 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:06.227936 30273 server_base.cc:1034] running on GCE node
I20260620 12:14:06.228868 30273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:06.230417 30273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:06.231643 30273 hybrid_clock.cc:648] HybridClock initialized: now 1781957646231609 us; error 69 us; skew 500 ppm
I20260620 12:14:06.239931 30273 webserver.cc:460] Webserver started at http://127.24.141.62:37793/ using document root <none> and password file <none>
I20260620 12:14:06.240733 30273 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:06.240836 30273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:06.241101 30273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:06.243675 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "93c02baca8424062a6fc1854a2ead540"
format_stamp: "Formatted at 2026-06-20 12:14:06 on dist-test-slave-rh5v"
server_key: "28e9533f0b5714826685b5de967e0edf"
server_key_iv: "3351ed2868bc700f067bc44baaba544c"
server_key_version: "encryptionkey@0"
I20260620 12:14:06.244398 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "93c02baca8424062a6fc1854a2ead540"
format_stamp: "Formatted at 2026-06-20 12:14:06 on dist-test-slave-rh5v"
server_key: "28e9533f0b5714826685b5de967e0edf"
server_key_iv: "3351ed2868bc700f067bc44baaba544c"
server_key_version: "encryptionkey@0"
I20260620 12:14:06.250629 30273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 12:14:06.254578 30287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:06.256625 30273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:14:06.256855 30273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "93c02baca8424062a6fc1854a2ead540"
format_stamp: "Formatted at 2026-06-20 12:14:06 on dist-test-slave-rh5v"
server_key: "28e9533f0b5714826685b5de967e0edf"
server_key_iv: "3351ed2868bc700f067bc44baaba544c"
server_key_version: "encryptionkey@0"
I20260620 12:14:06.257022 30273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:06.268718 30273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:06.269687 30273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:06.269905 30273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:06.280736 30273 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:36931
I20260620 12:14:06.280753 30339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:36931 every 8 connection(s)
I20260620 12:14:06.282032 30273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:06.286813 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30273
I20260620 12:14:06.286957 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:06.287305 25140 external_mini_cluster.cc:1428] Setting key 02c37915217d3ea84caf9ff4bc5424f5
I20260620 12:14:06.287333 30340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:06.297098 30340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540: Bootstrap starting.
I20260620 12:14:06.302076 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:06.289367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.24.141.62:36931)
0620 12:14:06.290451 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:06.290475 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:06.290536 (+ 61us) server_negotiation.cc:407] Connection header received
0620 12:14:06.291673 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:06.291706 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:06.292383 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:06.292961 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:06.293681 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.295304 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.296600 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.297481 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.297696 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:06.299007 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:06.299014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:06.299037 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:06.299683 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:06.299780 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:06.299815 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:06.299943 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:06.300268 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 12:14:06.300665 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":227,"threads_started":1}
I20260620 12:14:06.302263 30340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:06.303287 30340 log.cc:824] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:06.306143 30340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540: No bootstrap required, opened a new log
I20260620 12:14:06.307138 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:14:06.310541 30340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c02baca8424062a6fc1854a2ead540" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 36931 } }
I20260620 12:14:06.310902 30340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:06.310950 30340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c02baca8424062a6fc1854a2ead540, State: Initialized, Role: FOLLOWER
I20260620 12:14:06.311501 30340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c02baca8424062a6fc1854a2ead540" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 36931 } }
I20260620 12:14:06.311662 30340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:06.311714 30340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:06.311810 30340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:06.402563 30340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c02baca8424062a6fc1854a2ead540" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 36931 } }
I20260620 12:14:06.403306 30340 leader_election.cc:302] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c02baca8424062a6fc1854a2ead540; no voters:
I20260620 12:14:06.403805 30340 leader_election.cc:288] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:06.404042 30345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:06.404397 30345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [term 1 LEADER]: Becoming Leader. State: Replica: 93c02baca8424062a6fc1854a2ead540, State: Running, Role: LEADER
I20260620 12:14:06.404897 30345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c02baca8424062a6fc1854a2ead540" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 36931 } }
I20260620 12:14:06.405833 30340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:06.407035 30347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c02baca8424062a6fc1854a2ead540. Latest consensus state: current_term: 1 leader_uuid: "93c02baca8424062a6fc1854a2ead540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c02baca8424062a6fc1854a2ead540" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 36931 } } }
I20260620 12:14:06.407244 30347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:06.407774 30354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:06.407701 30346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93c02baca8424062a6fc1854a2ead540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c02baca8424062a6fc1854a2ead540" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 36931 } } }
I20260620 12:14:06.407873 30346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:06.411885 30354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:06.420343 30354 catalog_manager.cc:1347] Generated new cluster ID: 30696caa2a8a48d799df4d3d08e43941
I20260620 12:14:06.420461 30354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:06.461369 30354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:06.462939 30354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:06.471442 30354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93c02baca8424062a6fc1854a2ead540: Generated new TSK 0
I20260620 12:14:06.472369 30354 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 12:14:06.509846 30370 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 12:14:06.510058 30370 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 12:14:06.510821 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:06.488519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.24.141.62:36931)
0620 12:14:06.488727 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:06.488733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:06.488981 (+ 248us) server_negotiation.cc:407] Connection header received
0620 12:14:06.489442 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:06.489447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:06.489582 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:06.489765 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:14:06.490353 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.491294 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.492389 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.492852 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.493002 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:06.493983 (+ 981us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:14:06.510255 (+ 16272us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 12:14:06.510644 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55476: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 12:14:06.510962 30343 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55476: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 12:14:06.512416 30369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:06.488343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:36931 (local address 127.0.0.1:55476)
0620 12:14:06.488865 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:06.488882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:06.489107 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:06.489862 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:06.489866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:06.489878 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:14:06.490170 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:06.490206 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.491491 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.491494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:06.492222 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:06.492231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.493223 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.493227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:06.493468 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:06.493612 (+ 144us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:14:06.510649 (+ 17037us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 12:14:06.510951 (+ 302us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.141.62:36931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":215,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.141.62:36931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 12:14:06.513900 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30273
2026-06-20T12:14:06Z chronyd exiting
I20260620 12:14:06.553964 25140 test_util.cc:175] -----------------------------------------------
I20260620 12:14:06.554085 25140 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957618728254-25140-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (514 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 12:14:06.555505 25140 mini_oidc.cc:152] Starting JWKS server
I20260620 12:14:06.556466 25140 webserver.cc:460] Webserver started at https://127.0.0.1:44273/ using document root <none> and password file <none>
I20260620 12:14:06.556571 25140 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:14:06.556811 25140 webserver.cc:460] Webserver started at http://127.0.0.1:43929/ using document root <none> and password file <none>
2026-06-20T12:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:06Z Disabled control of system clock
I20260620 12:14:06.569650 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:34117
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:34117
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44273/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/testchainca.cert with env {}
W20260620 12:14:06.709764 30376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:06.710098 30376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:06.710142 30376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:06.714605 30376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:06.714702 30376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:06.714716 30376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:06.714731 30376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:06.714747 30376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:14:06.714763 30376 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:14:06.714776 30376 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44273/jwks.json
I20260620 12:14:06.720916 30376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:34117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44273/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:06.722271 30376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:06.723443 30376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:06.734228 30384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:06.734364 30382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:06.734215 30381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:06.734583 30376 server_base.cc:1034] running on GCE node
I20260620 12:14:06.735395 30376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:06.736750 30376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:06.737989 30376 hybrid_clock.cc:648] HybridClock initialized: now 1781957646737954 us; error 82 us; skew 500 ppm
I20260620 12:14:06.740324 30376 webserver.cc:460] Webserver started at http://127.24.141.62:44069/ using document root <none> and password file <none>
I20260620 12:14:06.741042 30376 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:06.741174 30376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:06.741531 30376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:06.744231 30376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0"
format_stamp: "Formatted at 2026-06-20 12:14:06 on dist-test-slave-rh5v"
server_key: "72a7814ac475f98dced22f3204bf427c"
server_key_iv: "5602ea9fe02fca3fc78d493bbd65f332"
server_key_version: "encryptionkey@0"
I20260620 12:14:06.744946 30376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0"
format_stamp: "Formatted at 2026-06-20 12:14:06 on dist-test-slave-rh5v"
server_key: "72a7814ac475f98dced22f3204bf427c"
server_key_iv: "5602ea9fe02fca3fc78d493bbd65f332"
server_key_version: "encryptionkey@0"
I20260620 12:14:06.750524 30376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 12:14:06.754169 30390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:06.756232 30376 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:14:06.756451 30376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0"
format_stamp: "Formatted at 2026-06-20 12:14:06 on dist-test-slave-rh5v"
server_key: "72a7814ac475f98dced22f3204bf427c"
server_key_iv: "5602ea9fe02fca3fc78d493bbd65f332"
server_key_version: "encryptionkey@0"
I20260620 12:14:06.756626 30376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:06.807680 30376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:06.808485 30376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:06.808678 30376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:06.816588 30376 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:34117
I20260620 12:14:06.816619 30442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:34117 every 8 connection(s)
I20260620 12:14:06.817715 30376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:06.820559 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30376
I20260620 12:14:06.820690 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:06.820969 25140 external_mini_cluster.cc:1428] Setting key 588dab60ee5fd3a7e4f805182e956856
I20260620 12:14:06.822325 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:06.830014 30443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0: Bootstrap starting.
I20260620 12:14:06.831792 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:06.822647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.24.141.62:34117)
0620 12:14:06.823628 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:06.823648 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:06.823697 (+ 49us) server_negotiation.cc:407] Connection header received
0620 12:14:06.824463 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:06.824486 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:06.824986 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:06.825406 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:06.825895 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.826954 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.827946 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.828526 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.828696 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:06.829661 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:06.829667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:06.829685 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:06.830168 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:06.830253 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:06.830284 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:06.830394 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:06.830549 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 12:14:06.830893 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":217,"threads_started":1}
I20260620 12:14:06.833417 30443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:06.834656 30443 log.cc:824] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:06.835767 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:14:06.837884 30443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0: No bootstrap required, opened a new log
I20260620 12:14:06.842370 30443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34117 } }
I20260620 12:14:06.842712 30443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:06.842765 30443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac2f97aa51a4ea19b3e57b8c92b24d0, State: Initialized, Role: FOLLOWER
I20260620 12:14:06.843307 30443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34117 } }
I20260620 12:14:06.843477 30443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:06.843547 30443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:06.843655 30443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:06.845032 30443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34117 } }
I20260620 12:14:06.845496 30443 leader_election.cc:302] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ac2f97aa51a4ea19b3e57b8c92b24d0; no voters:
I20260620 12:14:06.845870 30443 leader_election.cc:288] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:06.846140 30448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:06.846521 30448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5ac2f97aa51a4ea19b3e57b8c92b24d0, State: Running, Role: LEADER
I20260620 12:14:06.846989 30448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34117 } }
I20260620 12:14:06.847496 30443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:06.849013 30450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ac2f97aa51a4ea19b3e57b8c92b24d0. Latest consensus state: current_term: 1 leader_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34117 } } }
I20260620 12:14:06.849211 30450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:06.849655 30449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac2f97aa51a4ea19b3e57b8c92b24d0" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34117 } } }
I20260620 12:14:06.849854 30449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0 [sys.catalog]: This master's current role is: LEADER
W20260620 12:14:06.854293 30463 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:14:06.854416 30463 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:14:06.854521 30458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:06.855518 30458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:06.864203 30458 catalog_manager.cc:1347] Generated new cluster ID: 8bc0beb61ab84a19b9dfe2cc137dbe4d
I20260620 12:14:06.864358 30458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:06.877621 30458 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:06.879217 30458 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:06.889025 30458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ac2f97aa51a4ea19b3e57b8c92b24d0: Generated new TSK 0
I20260620 12:14:06.890369 30458 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 12:14:06.925243 30473 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 12:14:06.925366 30473 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 12:14:06.926316 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:06.908036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.24.141.62:34117)
0620 12:14:06.908233 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:06.908238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:06.908355 (+ 117us) server_negotiation.cc:407] Connection header received
0620 12:14:06.908563 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:06.908568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:06.908655 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:06.908830 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:14:06.909219 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.910085 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.911344 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.911745 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.911891 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:06.912318 (+ 427us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:14:06.925750 (+ 13432us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 12:14:06.926119 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 12:14:06.926424 30446 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 12:14:06.926651 30472 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:06.907863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34117 (local address 127.0.0.1:34530)
0620 12:14:06.908218 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:06.908235 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:06.908390 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:06.908873 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:06.908877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:06.908886 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:14:06.909033 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:06.909039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.910291 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.910295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:06.911138 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:06.911152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:06.911939 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:06.911943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:06.912045 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:06.912082 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:14:06.926099 (+ 14017us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 12:14:06.926348 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.141.62:34117: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":129,"threads_started":1}
I20260620 12:14:06.927423 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30376
2026-06-20T12:14:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (414 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 12:14:06.969738 25140 mini_oidc.cc:152] Starting JWKS server
I20260620 12:14:06.970693 25140 webserver.cc:460] Webserver started at https://127.0.0.1:44209/ using document root <none> and password file <none>
I20260620 12:14:06.970793 25140 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:14:06.971110 25140 webserver.cc:460] Webserver started at http://127.0.0.1:35405/ using document root <none> and password file <none>
2026-06-20T12:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:06Z Disabled control of system clock
I20260620 12:14:06.982806 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33955
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:33955
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 12:14:07.123121 30479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:07.123469 30479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:07.123529 30479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:07.128155 30479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:07.128266 30479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:07.128293 30479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:07.128315 30479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:07.128334 30479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:14:07.128355 30479 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:14:07.128373 30479 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44209/jwks.json
I20260620 12:14:07.134159 30479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:33955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:07.135475 30479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:07.137140 30479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:07.147233 30485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:07.147221 30484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:07.147221 30487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:07.147306 30479 server_base.cc:1034] running on GCE node
I20260620 12:14:07.148272 30479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:07.149711 30479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:07.150954 30479 hybrid_clock.cc:648] HybridClock initialized: now 1781957647150926 us; error 50 us; skew 500 ppm
I20260620 12:14:07.153455 30479 webserver.cc:460] Webserver started at http://127.24.141.62:35315/ using document root <none> and password file <none>
I20260620 12:14:07.154135 30479 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:07.154214 30479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:07.154433 30479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:07.156781 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "ac96fd04b88941eea43aca820532e3c6"
format_stamp: "Formatted at 2026-06-20 12:14:07 on dist-test-slave-rh5v"
server_key: "763df9aafb338727f95dfa6d9a87b84d"
server_key_iv: "3dfa25d2901ab1cff98db62e9083832e"
server_key_version: "encryptionkey@0"
I20260620 12:14:07.157498 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "ac96fd04b88941eea43aca820532e3c6"
format_stamp: "Formatted at 2026-06-20 12:14:07 on dist-test-slave-rh5v"
server_key: "763df9aafb338727f95dfa6d9a87b84d"
server_key_iv: "3dfa25d2901ab1cff98db62e9083832e"
server_key_version: "encryptionkey@0"
I20260620 12:14:07.162066 30479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:07.165297 30493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:07.167220 30479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:14:07.167433 30479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "ac96fd04b88941eea43aca820532e3c6"
format_stamp: "Formatted at 2026-06-20 12:14:07 on dist-test-slave-rh5v"
server_key: "763df9aafb338727f95dfa6d9a87b84d"
server_key_iv: "3dfa25d2901ab1cff98db62e9083832e"
server_key_version: "encryptionkey@0"
I20260620 12:14:07.167584 30479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:07.184957 30479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:07.185801 30479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:07.185997 30479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:07.193966 30479 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:33955
I20260620 12:14:07.194010 30545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:33955 every 8 connection(s)
I20260620 12:14:07.195097 30479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:07.199604 30546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:07.202549 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30479
I20260620 12:14:07.202677 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:07.203032 25140 external_mini_cluster.cc:1428] Setting key 5c17d380d119ad0dd377d047b0ad9267
I20260620 12:14:07.207089 30546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6: Bootstrap starting.
I20260620 12:14:07.210315 30546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:07.211383 30546 log.cc:824] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:07.213999 30546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6: No bootstrap required, opened a new log
I20260620 12:14:07.214447 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:07.204698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.24.141.62:33955)
0620 12:14:07.205519 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:07.205538 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:07.205583 (+ 45us) server_negotiation.cc:407] Connection header received
0620 12:14:07.206418 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:07.206438 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:07.206949 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:07.207362 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:07.207816 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.208857 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.210196 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.210803 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.210957 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:07.212077 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:07.212084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:07.212130 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:07.212617 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:07.212692 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:07.212717 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:07.212825 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:07.213000 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 12:14:07.213476 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":161,"threads_started":1}
I20260620 12:14:07.218078 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:14:07.218467 30546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96fd04b88941eea43aca820532e3c6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33955 } }
I20260620 12:14:07.218773 30546 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:07.218809 30546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac96fd04b88941eea43aca820532e3c6, State: Initialized, Role: FOLLOWER
I20260620 12:14:07.219370 30546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96fd04b88941eea43aca820532e3c6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33955 } }
I20260620 12:14:07.219534 30546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:07.219587 30546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:07.219697 30546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:07.220903 30546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96fd04b88941eea43aca820532e3c6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33955 } }
I20260620 12:14:07.221320 30546 leader_election.cc:302] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac96fd04b88941eea43aca820532e3c6; no voters:
I20260620 12:14:07.221680 30546 leader_election.cc:288] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:07.222059 30551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:07.222461 30551 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [term 1 LEADER]: Becoming Leader. State: Replica: ac96fd04b88941eea43aca820532e3c6, State: Running, Role: LEADER
I20260620 12:14:07.222904 30551 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96fd04b88941eea43aca820532e3c6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33955 } }
I20260620 12:14:07.223451 30546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:07.225188 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac96fd04b88941eea43aca820532e3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96fd04b88941eea43aca820532e3c6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33955 } } }
I20260620 12:14:07.225188 30553 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac96fd04b88941eea43aca820532e3c6. Latest consensus state: current_term: 1 leader_uuid: "ac96fd04b88941eea43aca820532e3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96fd04b88941eea43aca820532e3c6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33955 } } }
I20260620 12:14:07.225414 30553 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:07.225414 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:07.226011 30565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:07.230116 30565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:07.238257 30565 catalog_manager.cc:1347] Generated new cluster ID: 6b4f21dc5fda4fdaaf72559135bb8ef4
I20260620 12:14:07.238410 30565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:07.276919 30565 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:07.278137 30565 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:07.290730 30565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac96fd04b88941eea43aca820532e3c6: Generated new TSK 0
I20260620 12:14:07.291795 30565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:14:07.314666 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:07.308577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.24.141.62:33955)
0620 12:14:07.308755 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:07.308760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:07.308852 (+ 92us) server_negotiation.cc:407] Connection header received
0620 12:14:07.309120 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:07.309124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:07.309211 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:07.309375 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:14:07.309676 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.310527 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.311538 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.311884 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.312002 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:07.312408 (+ 406us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:14:07.314213 (+ 1805us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 12:14:07.314498 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 12:14:07.314765 30549 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 12:14:07.316243 30575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:07.308411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33955 (local address 127.0.0.1:35974)
0620 12:14:07.308738 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:07.308753 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:07.308889 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:07.309416 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:07.309419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:07.309427 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:14:07.309557 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:07.309564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.310668 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.310672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:07.311419 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:07.311428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.312063 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.312067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:07.312145 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:07.312177 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:14:07.314425 (+ 2248us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 12:14:07.316010 (+ 1585us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.141.62:33955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1339,"thread_start_us":113,"threads_started":1}
I20260620 12:14:07.317142 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30479
E20260620 12:14:07.325565 30576 webserver.cc:533] Webserver: sslize failed:
E20260620 12:14:07.325670 30576 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T12:14:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (387 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T12:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:07Z Disabled control of system clock
I20260620 12:14:07.368564 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38423
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:38423
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/testchainca.cert with env {}
W20260620 12:14:07.506662 30580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:07.507042 30580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:07.507090 30580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:07.511657 30580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:07.511762 30580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:07.511777 30580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:07.511794 30580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:07.511809 30580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:14:07.511827 30580 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:14:07.511839 30580 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 12:14:07.518370 30580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:38423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:07.519881 30580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:07.521201 30580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:07.532060 30585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:07.532060 30588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:07.532066 30586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:07.532475 30580 server_base.cc:1034] running on GCE node
I20260620 12:14:07.533527 30580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:07.535077 30580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:07.536278 30580 hybrid_clock.cc:648] HybridClock initialized: now 1781957647536263 us; error 43 us; skew 500 ppm
I20260620 12:14:07.538995 30580 webserver.cc:460] Webserver started at http://127.24.141.62:32941/ using document root <none> and password file <none>
I20260620 12:14:07.539781 30580 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:07.539863 30580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:07.540098 30580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:07.542769 30580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "16dfc41345a14dddbb75e9e117ddc5af"
format_stamp: "Formatted at 2026-06-20 12:14:07 on dist-test-slave-rh5v"
server_key: "75676cae3f3728212d66fb633cfc1ccf"
server_key_iv: "2759cf77636e65fffe40b4d01162be9a"
server_key_version: "encryptionkey@0"
I20260620 12:14:07.543545 30580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "16dfc41345a14dddbb75e9e117ddc5af"
format_stamp: "Formatted at 2026-06-20 12:14:07 on dist-test-slave-rh5v"
server_key: "75676cae3f3728212d66fb633cfc1ccf"
server_key_iv: "2759cf77636e65fffe40b4d01162be9a"
server_key_version: "encryptionkey@0"
I20260620 12:14:07.548273 30580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:14:07.551878 30594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:07.554075 30580 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 12:14:07.554270 30580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "16dfc41345a14dddbb75e9e117ddc5af"
format_stamp: "Formatted at 2026-06-20 12:14:07 on dist-test-slave-rh5v"
server_key: "75676cae3f3728212d66fb633cfc1ccf"
server_key_iv: "2759cf77636e65fffe40b4d01162be9a"
server_key_version: "encryptionkey@0"
I20260620 12:14:07.554425 30580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:07.579622 30580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:07.580559 30580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:07.580765 30580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:07.589421 30580 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:38423
I20260620 12:14:07.589491 30646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:38423 every 8 connection(s)
I20260620 12:14:07.590724 30580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:07.595337 30647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:07.598094 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30580
I20260620 12:14:07.598258 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:07.598559 25140 external_mini_cluster.cc:1428] Setting key 5f4d4684151d020b074cd14916d636e5
I20260620 12:14:07.603878 30647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af: Bootstrap starting.
I20260620 12:14:07.606992 30647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:07.608012 30647 log.cc:824] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:07.611074 30647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af: No bootstrap required, opened a new log
I20260620 12:14:07.611218 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:07.600548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.24.141.62:38423)
0620 12:14:07.601515 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:07.601540 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:07.601596 (+ 56us) server_negotiation.cc:407] Connection header received
0620 12:14:07.602503 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:07.602532 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:07.603137 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:07.603567 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:07.604099 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.605648 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.606703 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.607448 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.607647 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:07.608758 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:07.608764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:07.608780 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:07.609372 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:07.609462 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:07.609492 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:07.609617 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:07.609786 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 12:14:07.610146 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":138,"threads_started":1}
I20260620 12:14:07.614933 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:14:07.615345 30647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dfc41345a14dddbb75e9e117ddc5af" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38423 } }
I20260620 12:14:07.615721 30647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:07.615772 30647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16dfc41345a14dddbb75e9e117ddc5af, State: Initialized, Role: FOLLOWER
I20260620 12:14:07.616451 30647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dfc41345a14dddbb75e9e117ddc5af" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38423 } }
I20260620 12:14:07.616668 30647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:07.616760 30647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:07.616882 30647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:07.618088 30647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dfc41345a14dddbb75e9e117ddc5af" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38423 } }
I20260620 12:14:07.618577 30647 leader_election.cc:302] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16dfc41345a14dddbb75e9e117ddc5af; no voters:
I20260620 12:14:07.618974 30647 leader_election.cc:288] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:07.619176 30652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:07.619529 30652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [term 1 LEADER]: Becoming Leader. State: Replica: 16dfc41345a14dddbb75e9e117ddc5af, State: Running, Role: LEADER
I20260620 12:14:07.619978 30652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dfc41345a14dddbb75e9e117ddc5af" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38423 } }
I20260620 12:14:07.620464 30647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:07.622054 30653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16dfc41345a14dddbb75e9e117ddc5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dfc41345a14dddbb75e9e117ddc5af" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38423 } } }
I20260620 12:14:07.622175 30654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16dfc41345a14dddbb75e9e117ddc5af. Latest consensus state: current_term: 1 leader_uuid: "16dfc41345a14dddbb75e9e117ddc5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dfc41345a14dddbb75e9e117ddc5af" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38423 } } }
I20260620 12:14:07.622279 30653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:07.622300 30654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:07.622839 30663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:07.626721 30663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:07.634526 30663 catalog_manager.cc:1347] Generated new cluster ID: 0c39499fbc35477cb72b964ee9d23d21
I20260620 12:14:07.634614 30663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:07.647356 30663 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:07.648722 30663 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:07.656656 30663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16dfc41345a14dddbb75e9e117ddc5af: Generated new TSK 0
I20260620 12:14:07.657577 30663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:14:07.710835 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:07.666224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.24.141.62:38423)
0620 12:14:07.666409 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:07.666415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:07.666723 (+ 308us) server_negotiation.cc:407] Connection header received
0620 12:14:07.666997 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:07.667000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:07.667069 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:07.667180 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:14:07.667534 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.668180 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.669658 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.669932 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.670043 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:07.670523 (+ 480us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:14:07.710282 (+ 39759us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 12:14:07.710674 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55534: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 12:14:07.710943 30650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55534: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 12:14:07.711158 30676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:07.666007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:38423 (local address 127.0.0.1:55534)
0620 12:14:07.666575 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:07.666594 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:07.666791 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:07.667204 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:07.667207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:07.667216 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:14:07.667388 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:07.667396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.668315 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.668319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:07.669462 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:07.669478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:07.670091 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:07.670095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:07.670206 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:07.670250 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:14:07.710632 (+ 40382us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 12:14:07.710901 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.141.62:38423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":138,"threads_started":1}
I20260620 12:14:07.712218 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30580
2026-06-20T12:14:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (375 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (146 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (148 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:08 dist-test-slave-rh5v krb5kdc[30688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:08 dist-test-slave-rh5v krb5kdc[30688](info): set up 2 sockets
Jun 20 12:14:08 dist-test-slave-rh5v krb5kdc[30688](info): commencing operation
krb5kdc: starting...
W20260620 12:14:08.985957 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:09 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:14:09.128764 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:42101
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:09.262943 30701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:09.263290 30701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:09.263355 30701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:09.268718 30701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:09.268829 30701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:09.268860 30701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:09.268884 30701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:09.268904 30701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:09.275249 30701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:09.276566 30701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:09.277822 30701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:09.288159 30706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:09.288175 30707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:09.288322 30701 server_base.cc:1034] running on GCE node
W20260620 12:14:09.288175 30709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:09.289269 30701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:09.290686 30701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:09.291939 30701 hybrid_clock.cc:648] HybridClock initialized: now 1781957649291907 us; error 55 us; skew 500 ppm
Jun 20 12:14:09 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:09.295328 30701 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:14:09.296514 30701 webserver.cc:460] Webserver started at http://127.24.141.62:38053/ using document root <none> and password file <none>
I20260620 12:14:09.297268 30701 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:09.297353 30701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:09.297556 30701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:09.300285 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "e1a8df8da9c34e619153de7b89d53b08"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "110af0865f5dec8b50f5b521a6963761"
server_key_iv: "b191253d0316a79256d184b2646ba1c9"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.300949 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "e1a8df8da9c34e619153de7b89d53b08"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "110af0865f5dec8b50f5b521a6963761"
server_key_iv: "b191253d0316a79256d184b2646ba1c9"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.305753 30701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:14:09.309196 30716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:09.310983 30701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:09.311177 30701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "e1a8df8da9c34e619153de7b89d53b08"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "110af0865f5dec8b50f5b521a6963761"
server_key_iv: "b191253d0316a79256d184b2646ba1c9"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.311335 30701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:09.330484 30701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:09.334555 30701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:09.334786 30701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:09.343487 30701 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:42101
I20260620 12:14:09.343532 30768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:42101 every 8 connection(s)
I20260620 12:14:09.344703 30701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:09.347980 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30701
I20260620 12:14:09.348109 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:09.348428 25140 external_mini_cluster.cc:1428] Setting key 3b20daac7577c6a17adf9f0b8cbc1d4b
I20260620 12:14:09.348860 30769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:09.357110 30769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08: Bootstrap starting.
I20260620 12:14:09.360379 30769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08: Neither blocks nor log segments found. Creating new log.
Jun 20 12:14:09 dist-test-slave-rh5v krb5kdc[30688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:09.361500 30769 log.cc:824] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:09.364840 30769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08: No bootstrap required, opened a new log
I20260620 12:14:09.368386 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:09.350290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.24.141.62:42101)
0620 12:14:09.350838 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:09.350861 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:09.351122 (+ 261us) server_negotiation.cc:407] Connection header received
0620 12:14:09.352420 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:09.352447 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:09.353242 (+ 795us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:09.353690 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:09.355139 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.356301 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.357321 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.358145 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.358373 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:09.361669 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:09.361678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:09.361699 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:09.361746 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:09.364035 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:09.364758 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:09.364765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:09.364772 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:09.364957 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:09.365338 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:09.365341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:09.365343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:09.365977 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:09.366201 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:09.366693 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 12:14:09.367101 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":164,"threads_started":1}
I20260620 12:14:09.370155 30769 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a8df8da9c34e619153de7b89d53b08" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42101 } }
I20260620 12:14:09.370512 30769 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:09.370584 30769 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a8df8da9c34e619153de7b89d53b08, State: Initialized, Role: FOLLOWER
I20260620 12:14:09.371176 30769 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a8df8da9c34e619153de7b89d53b08" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42101 } }
I20260620 12:14:09.371325 30769 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:09.371397 30769 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:09.371500 30769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:09.372669 30769 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a8df8da9c34e619153de7b89d53b08" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42101 } }
I20260620 12:14:09.373028 30769 leader_election.cc:302] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1a8df8da9c34e619153de7b89d53b08; no voters:
I20260620 12:14:09.373430 30769 leader_election.cc:288] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:09.373704 30774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:09.374162 30774 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [term 1 LEADER]: Becoming Leader. State: Replica: e1a8df8da9c34e619153de7b89d53b08, State: Running, Role: LEADER
I20260620 12:14:09.374913 30769 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:09.374919 30774 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a8df8da9c34e619153de7b89d53b08" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42101 } }
I20260620 12:14:09.377362 30775 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1a8df8da9c34e619153de7b89d53b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a8df8da9c34e619153de7b89d53b08" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42101 } } }
I20260620 12:14:09.377977 30775 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:09.377643 30776 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1a8df8da9c34e619153de7b89d53b08. Latest consensus state: current_term: 1 leader_uuid: "e1a8df8da9c34e619153de7b89d53b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a8df8da9c34e619153de7b89d53b08" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42101 } } }
I20260620 12:14:09.378419 30776 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08 [sys.catalog]: This master's current role is: LEADER
W20260620 12:14:09.381675 30788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:14:09.381793 30788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:14:09.381916 30789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:09.383175 30789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:09.391286 30789 catalog_manager.cc:1347] Generated new cluster ID: 8741e2e4b705488c856256f7dcf2c1ce
I20260620 12:14:09.391407 30789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:09.412737 30789 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:09.413928 30789 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:09.422267 30789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1a8df8da9c34e619153de7b89d53b08: Generated new TSK 0
I20260620 12:14:09.423281 30789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:14:09.461547 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42101
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:09.609145 30795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:09.609468 30795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:09.609531 30795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:09.613957 30795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:09.614043 30795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:09.614130 30795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:09.621291 30795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:09.623525 30795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:09.625309 30795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:09.638545 30803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:09.638504 30801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:09.638550 30800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:09.639259 30795 server_base.cc:1034] running on GCE node
I20260620 12:14:09.639905 30795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:09.640846 30795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:09.642038 30795 hybrid_clock.cc:648] HybridClock initialized: now 1781957649642011 us; error 50 us; skew 500 ppm
Jun 20 12:14:09 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:09.645645 30795 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:14:09.646850 30795 webserver.cc:460] Webserver started at http://127.24.141.1:34331/ using document root <none> and password file <none>
I20260620 12:14:09.647548 30795 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:09.647616 30795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:09.647826 30795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:09.650688 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "df693e16060a48cd9b82714ce5640994"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "d02104754020e88c37566b4ac8fbb23b"
server_key_iv: "da1826729e31110ded6973d4327f60c4"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.651346 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "df693e16060a48cd9b82714ce5640994"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "d02104754020e88c37566b4ac8fbb23b"
server_key_iv: "da1826729e31110ded6973d4327f60c4"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.656345 30795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:14:09.659880 30810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:09.662029 30795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:09.662217 30795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "df693e16060a48cd9b82714ce5640994"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "d02104754020e88c37566b4ac8fbb23b"
server_key_iv: "da1826729e31110ded6973d4327f60c4"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.662364 30795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:09.685420 30795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:09.689949 30795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:09.690165 30795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:09.690917 30795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:09.692050 30795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:09.692108 30795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:09.692157 30795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:09.692171 30795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:09.703826 30795 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:37185
I20260620 12:14:09.703943 30923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:37185 every 8 connection(s)
I20260620 12:14:09.705096 30795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:09.714394 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30795
I20260620 12:14:09.714540 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:09.714834 25140 external_mini_cluster.cc:1428] Setting key fa0b2e5f6a0ac2a61d7c4160e2d19811
Jun 20 12:14:09 dist-test-slave-rh5v krb5kdc[30688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:09.724459 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:09.707870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:41509 (local address 127.24.141.62:42101)
0620 12:14:09.708176 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:09.708182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:09.709459 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 12:14:09.711262 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:09.711265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:09.711331 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:09.711644 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:09.713260 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.713886 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.715560 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.715881 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.716005 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:09.719553 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:09.719557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:09.719559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:09.719601 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:09.721087 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:09.722136 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:09.722140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:09.722142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:09.722265 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:09.722818 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:09.722822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:09.722824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:09.723066 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:09.723169 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:09.724130 (+ 961us) server_negotiation.cc:299] Negotiation successful
0620 12:14:09.724308 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:14:09.726027 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:09.708565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:42101 (local address 127.24.141.1:41509)
0620 12:14:09.709246 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:09.709308 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:09.710935 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:09.711800 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:09.711810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:09.712285 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:09.712958 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:09.712975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.714076 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.714080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:09.715340 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:09.715355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.716128 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.716132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:09.716365 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:09.717162 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:09.717184 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:09.719295 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:09.721335 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:09.721347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:09.721378 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:09.721914 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:09.722382 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:09.722385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:09.722387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:09.722638 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:09.723193 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:09.723202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:09.723695 (+ 493us) client_negotiation.cc:769] Sending connection context
0620 12:14:09.723987 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 12:14:09.724480 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":181,"threads_started":1}
I20260620 12:14:09.728117 30924 heartbeater.cc:347] Connected to a master server at 127.24.141.62:42101
I20260620 12:14:09.728632 30924 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:14:09.729622 30924 heartbeater.cc:511] Master 127.24.141.62:42101 requested a full tablet report, sending...
I20260620 12:14:09.732035 30733 ts_manager.cc:194] Registered new tserver with Master: df693e16060a48cd9b82714ce5640994 (127.24.141.1:37185)
I20260620 12:14:09.733539 30733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:41509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:14:09.748236 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42101
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:09.871451 30929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:09.871920 30929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:09.872125 30929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:09.877972 30929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:09.878078 30929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:09.878173 30929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:09.884268 30929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:09.885625 30929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:09.886785 30929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:09.897943 30934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:09.897933 30935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:09.898411 30929 server_base.cc:1034] running on GCE node
W20260620 12:14:09.898599 30937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:09.899048 30929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:09.899777 30929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:09.900996 30929 hybrid_clock.cc:648] HybridClock initialized: now 1781957649900979 us; error 37 us; skew 500 ppm
Jun 20 12:14:09 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:09.904273 30929 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:14:09.905469 30929 webserver.cc:460] Webserver started at http://127.24.141.2:37025/ using document root <none> and password file <none>
I20260620 12:14:09.906138 30929 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:09.906215 30929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:09.906452 30929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:09.908797 30929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "6d093516713841ad83c7724e363638b3"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "2bd9744c49bab4bb101e5af4f5a29999"
server_key_iv: "4150b7a61ce122c32965df77cf338ae2"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.909498 30929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "6d093516713841ad83c7724e363638b3"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "2bd9744c49bab4bb101e5af4f5a29999"
server_key_iv: "4150b7a61ce122c32965df77cf338ae2"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.914269 30929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:14:09.918058 30944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:09.920109 30929 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:09.920290 30929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "6d093516713841ad83c7724e363638b3"
format_stamp: "Formatted at 2026-06-20 12:14:09 on dist-test-slave-rh5v"
server_key: "2bd9744c49bab4bb101e5af4f5a29999"
server_key_iv: "4150b7a61ce122c32965df77cf338ae2"
server_key_version: "encryptionkey@0"
I20260620 12:14:09.920452 30929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:09.945299 30929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:09.948863 30929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:09.949115 30929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:09.949802 30929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:09.950790 30929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:09.950845 30929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:09.950901 30929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:09.950930 30929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:09.961876 30929 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:36299
I20260620 12:14:09.961923 31057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:36299 every 8 connection(s)
I20260620 12:14:09.962965 30929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:09.967195 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 30929
I20260620 12:14:09.967361 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:09.967710 25140 external_mini_cluster.cc:1428] Setting key 01f35e6663909e913a3470dedf88b3b3
Jun 20 12:14:09 dist-test-slave-rh5v krb5kdc[30688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:14:09.982311 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:09.965670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:34601 (local address 127.24.141.62:42101)
0620 12:14:09.965884 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:09.965889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:09.967109 (+ 1220us) server_negotiation.cc:407] Connection header received
0620 12:14:09.968811 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:09.968815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:09.968890 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:09.968968 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:09.970866 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.971763 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.973291 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.973685 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.973834 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:09.977125 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:09.977130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:09.977134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:09.977184 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:09.979141 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:09.980008 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:09.980013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:09.980014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:09.980130 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:09.980736 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:09.980739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:09.980740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:09.981016 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:09.981196 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:09.982000 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 12:14:09.982166 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 12:14:09.983886 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:09.966129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:42101 (local address 127.24.141.2:34601)
0620 12:14:09.966910 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:09.966979 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:09.968521 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:09.969322 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:09.969333 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:09.969921 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:09.970642 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:09.970665 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.971998 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.972002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:09.973122 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:09.973138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:09.973900 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:09.973904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:09.974086 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:09.974869 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:09.974898 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:09.976798 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:09.979321 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:09.979330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:09.979356 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:09.979809 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:09.980291 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:09.980294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:09.980295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:09.980564 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:09.981200 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:09.981210 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:09.981641 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 12:14:09.981917 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 12:14:09.982425 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":186,"threads_started":1}
I20260620 12:14:09.985523 31058 heartbeater.cc:347] Connected to a master server at 127.24.141.62:42101
I20260620 12:14:09.985846 31058 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:09.986534 31058 heartbeater.cc:511] Master 127.24.141.62:42101 requested a full tablet report, sending...
I20260620 12:14:09.988178 30733 ts_manager.cc:194] Registered new tserver with Master: 6d093516713841ad83c7724e363638b3 (127.24.141.2:36299)
I20260620 12:14:09.988888 30733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:34601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:14:10.001924 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42101
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:10.141598 31063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:10.141985 31063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:10.142043 31063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:10.147306 31063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:10.147396 31063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:10.147459 31063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:10.153992 31063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:10.155215 31063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:10.156347 31063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:10.168758 31068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:10.168761 31069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:10.168761 31071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:10.169409 31063 server_base.cc:1034] running on GCE node
I20260620 12:14:10.169927 31063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:10.170996 31063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:10.172219 31063 hybrid_clock.cc:648] HybridClock initialized: now 1781957650172192 us; error 53 us; skew 500 ppm
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.176119 31063 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:14:10.177814 31063 webserver.cc:460] Webserver started at http://127.24.141.3:46113/ using document root <none> and password file <none>
I20260620 12:14:10.178504 31063 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:10.178589 31063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:10.178805 31063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:10.181865 31063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "0095a2372d3648e48b60543870aa66f2"
format_stamp: "Formatted at 2026-06-20 12:14:10 on dist-test-slave-rh5v"
server_key: "95cd48b48efccdbe8d0c0bcc90adbc55"
server_key_iv: "baccd5905780d34e6c17ac5613518db8"
server_key_version: "encryptionkey@0"
I20260620 12:14:10.182754 31063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "0095a2372d3648e48b60543870aa66f2"
format_stamp: "Formatted at 2026-06-20 12:14:10 on dist-test-slave-rh5v"
server_key: "95cd48b48efccdbe8d0c0bcc90adbc55"
server_key_iv: "baccd5905780d34e6c17ac5613518db8"
server_key_version: "encryptionkey@0"
I20260620 12:14:10.188405 31063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:14:10.192023 31078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:10.193930 31063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:10.194141 31063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "0095a2372d3648e48b60543870aa66f2"
format_stamp: "Formatted at 2026-06-20 12:14:10 on dist-test-slave-rh5v"
server_key: "95cd48b48efccdbe8d0c0bcc90adbc55"
server_key_iv: "baccd5905780d34e6c17ac5613518db8"
server_key_version: "encryptionkey@0"
I20260620 12:14:10.194314 31063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:10.229638 31063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:10.233825 31063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:10.234066 31063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:10.234834 31063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:10.236011 31063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:10.236068 31063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:10.236115 31063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:10.236130 31063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:10.247428 31063 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:38925
I20260620 12:14:10.247520 31191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:38925 every 8 connection(s)
I20260620 12:14:10.248641 31063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:10.252521 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 31063
I20260620 12:14:10.252717 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:10.253115 25140 external_mini_cluster.cc:1428] Setting key bfe7629ea4d6e794a72621e6ba87967f
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:10.265666 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:10.251196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:39485 (local address 127.24.141.62:42101)
0620 12:14:10.251357 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:10.251361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:10.252824 (+ 1463us) server_negotiation.cc:407] Connection header received
0620 12:14:10.254805 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:10.254809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:10.254876 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:10.254957 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:10.256483 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.257035 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.258322 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.258542 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.258642 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:10.261556 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:10.261559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:10.261562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:10.261598 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:10.263002 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.263932 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.263935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.263936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.264027 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.264412 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.264415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.264416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.264625 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:10.264719 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:10.265335 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 12:14:10.265491 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:14:10.266749 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:10.251654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:42101 (local address 127.24.141.3:39485)
0620 12:14:10.252557 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:10.252659 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:10.254521 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:10.255204 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:10.255213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:10.255664 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:10.256316 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:10.256334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.257203 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.257206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:10.258190 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:10.258200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.258702 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.258706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:10.258891 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:10.259651 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:10.259671 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:10.261334 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:10.263273 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:10.263282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:10.263305 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:10.263786 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:10.264136 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:10.264139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:10.264141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:10.264314 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:10.264733 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:10.264741 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:10.265037 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 12:14:10.265284 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 12:14:10.265678 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":189,"threads_started":1}
I20260620 12:14:10.268369 31192 heartbeater.cc:347] Connected to a master server at 127.24.141.62:42101
I20260620 12:14:10.268712 31192 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:10.269413 31192 heartbeater.cc:511] Master 127.24.141.62:42101 requested a full tablet report, sending...
I20260620 12:14:10.270969 30733 ts_manager.cc:194] Registered new tserver with Master: 0095a2372d3648e48b60543870aa66f2 (127.24.141.3:38925)
I20260620 12:14:10.271726 30733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:39485
I20260620 12:14:10.281229 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/test-admin.keytab.
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.298055 25140 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 12:14:10.301889 31199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:10.299918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.24.141.1:37185)
0620 12:14:10.300386 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:10.300394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:10.300429 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:14:10.300498 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:10.300505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:10.300758 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:10.300926 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:10.301685 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60062: BlockingRecv error: recv got EOF from 127.0.0.1:60062 (error 108)
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":161,"threads_started":1}
W20260620 12:14:10.302034 25140 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5cc' of type FILE
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.303175 25140 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.304188 25140 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:14:10.317639 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:10.305372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.24.141.1:37185)
0620 12:14:10.305521 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:10.305526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:10.305615 (+ 89us) server_negotiation.cc:407] Connection header received
0620 12:14:10.305774 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:10.305777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:10.305829 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:10.305911 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:10.306643 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.307559 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.308763 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.309403 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.309520 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:10.312666 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:10.312683 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:10.312706 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:10.312775 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:10.314881 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.315672 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.315680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.315681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.315801 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.316156 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.316159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.316160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.316594 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:10.316804 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:10.317210 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 12:14:10.317436 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 12:14:10.323127 31199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:10.321913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.24.141.1:37185)
0620 12:14:10.322077 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:10.322081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:10.322162 (+ 81us) server_negotiation.cc:407] Connection header received
0620 12:14:10.322303 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:10.322306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:10.322357 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:10.322470 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:10.322992 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60088: BlockingRecv error: recv got EOF from 127.0.0.1:60088 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 12:14:10.323468 25140 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5cc' of type FILE
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.324753 25140 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.325897 25140 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:14:10.337188 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:10.327037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.24.141.1:37185)
0620 12:14:10.327250 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:10.327254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:10.327370 (+ 116us) server_negotiation.cc:407] Connection header received
0620 12:14:10.327492 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:10.327495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:10.327551 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:10.327636 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:10.328260 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.328888 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.329867 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.330123 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.330222 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:10.333176 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:10.333180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:10.333183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:10.333218 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:10.334836 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.335576 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.335580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.335581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.335672 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.336118 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.336122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.336123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.336339 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:10.336535 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:10.336819 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 12:14:10.336962 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 12:14:10.341396 31199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:10.339980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.24.141.1:37185)
0620 12:14:10.340147 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:10.340150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:10.340244 (+ 94us) server_negotiation.cc:407] Connection header received
0620 12:14:10.340383 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:10.340386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:10.340435 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:10.340509 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:10.341278 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60112: BlockingRecv error: recv got EOF from 127.0.0.1:60112 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 12:14:10.341511 25140 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957618728254-25140-0/krb5kdc/krb5cc' of type FILE
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.342797 25140 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:10.343930 25140 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[30688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:14:10.353367 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:10.345139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.24.141.1:37185)
0620 12:14:10.345255 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:10.345259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:10.345450 (+ 191us) server_negotiation.cc:407] Connection header received
0620 12:14:10.345603 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:10.345606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:10.345652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:10.345717 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:10.346308 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.346855 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.347663 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:10.347888 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:10.347975 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:10.350382 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:10.350385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:10.350388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:10.350414 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:10.351718 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.352126 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.352130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.352132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.352209 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:10.352452 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:10.352455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:10.352457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:10.352645 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:10.352795 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:10.353008 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 12:14:10.353174 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:14:10.354635 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30795
I20260620 12:14:10.360786 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30929
I20260620 12:14:10.367759 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 31063
I20260620 12:14:10.374392 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 30701
2026-06-20T12:14:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2370 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[31212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[31212](info): set up 2 sockets
Jun 20 12:14:10 dist-test-slave-rh5v krb5kdc[31212](info): commencing operation
krb5kdc: starting...
W20260620 12:14:11.350093 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:11 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab.
Entry for principal oryx/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab.
I20260620 12:14:11.489431 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:34705
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab
--principal=oryx/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:34705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:11.623737 31225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:11.624136 31225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:11.624187 31225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:11.628887 31225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:11.628975 31225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:11.628995 31225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:11.629012 31225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:11.629027 31225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:11.635393 31225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:34705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab
--principal=oryx/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:34705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:11.636893 31225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:11.638227 31225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:11.648375 31230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:11.648469 31231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:11.648375 31233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:11.649156 31225 server_base.cc:1034] running on GCE node
I20260620 12:14:11.649972 31225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:11.651460 31225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:11.652706 31225 hybrid_clock.cc:648] HybridClock initialized: now 1781957651652673 us; error 54 us; skew 500 ppm
Jun 20 12:14:11 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:11.660157 31225 init.cc:377] Logged in from keytab as oryx/127.24.141.62@KRBTEST.COM (short username oryx)
I20260620 12:14:11.661434 31225 webserver.cc:460] Webserver started at http://127.24.141.62:46083/ using document root <none> and password file <none>
I20260620 12:14:11.662104 31225 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:11.662175 31225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:11.662400 31225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:11.664909 31225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "b49e4418976f48feace83ae97c424b7c"
format_stamp: "Formatted at 2026-06-20 12:14:11 on dist-test-slave-rh5v"
server_key: "0645f1fc29d28c1ecd44640634e1c6d8"
server_key_iv: "1c7abd13169195f38be5257ae934e27e"
server_key_version: "encryptionkey@0"
I20260620 12:14:11.665624 31225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "b49e4418976f48feace83ae97c424b7c"
format_stamp: "Formatted at 2026-06-20 12:14:11 on dist-test-slave-rh5v"
server_key: "0645f1fc29d28c1ecd44640634e1c6d8"
server_key_iv: "1c7abd13169195f38be5257ae934e27e"
server_key_version: "encryptionkey@0"
I20260620 12:14:11.670495 31225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:11.673770 31240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:11.675731 31225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:14:11.675932 31225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "b49e4418976f48feace83ae97c424b7c"
format_stamp: "Formatted at 2026-06-20 12:14:11 on dist-test-slave-rh5v"
server_key: "0645f1fc29d28c1ecd44640634e1c6d8"
server_key_iv: "1c7abd13169195f38be5257ae934e27e"
server_key_version: "encryptionkey@0"
I20260620 12:14:11.676074 31225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:11.691474 31225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:11.694936 31225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:11.695145 31225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:11.703199 31225 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:34705
I20260620 12:14:11.703542 31292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:34705 every 8 connection(s)
I20260620 12:14:11.704442 31225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:11.708366 31293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:11.708691 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 31225
I20260620 12:14:11.708843 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:11.709275 25140 external_mini_cluster.cc:1428] Setting key 2c6fdbd603f8a634e76e4e2c1ecbecf2
I20260620 12:14:11.717422 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: Bootstrap starting.
I20260620 12:14:11.720634 31293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: Neither blocks nor log segments found. Creating new log.
Jun 20 12:14:11 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:11.721625 31293 log.cc:824] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:11.724056 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: No bootstrap required, opened a new log
I20260620 12:14:11.727465 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:11.711206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.24.141.62:34705)
0620 12:14:11.711799 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:11.711818 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:11.711868 (+ 50us) server_negotiation.cc:407] Connection header received
0620 12:14:11.712767 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:11.712790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:11.713300 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:11.713721 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:11.714829 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:11.716007 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:11.717313 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:11.717991 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:11.718178 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:11.721448 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:11.721457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:11.721481 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:11.721528 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:11.723789 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:11.724447 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:11.724455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:11.724461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:11.724650 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:11.724986 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:11.724989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:11.724990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:11.725569 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:11.725776 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:11.726114 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 12:14:11.726492 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":173,"threads_started":1}
I20260620 12:14:11.728688 31293 raft_consensus.cc:348] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } }
I20260620 12:14:11.728971 31293 raft_consensus.cc:374] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:11.729018 31293 raft_consensus.cc:729] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49e4418976f48feace83ae97c424b7c, State: Initialized, Role: FOLLOWER
I20260620 12:14:11.729624 31293 consensus_queue.cc:260] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } }
I20260620 12:14:11.729779 31293 raft_consensus.cc:388] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:11.729849 31293 raft_consensus.cc:482] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:11.729954 31293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:11.731120 31293 raft_consensus.cc:504] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } }
I20260620 12:14:11.731500 31293 leader_election.cc:302] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b49e4418976f48feace83ae97c424b7c; no voters:
I20260620 12:14:11.731846 31293 leader_election.cc:288] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:11.731959 31298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:11.732147 31298 raft_consensus.cc:686] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 1 LEADER]: Becoming Leader. State: Replica: b49e4418976f48feace83ae97c424b7c, State: Running, Role: LEADER
I20260620 12:14:11.732501 31298 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } }
I20260620 12:14:11.733506 31293 sys_catalog.cc:564] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:11.734742 31300 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49e4418976f48feace83ae97c424b7c. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } }
I20260620 12:14:11.734942 31300 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:11.734704 31299 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } }
I20260620 12:14:11.735186 31299 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:11.735455 31307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:11.739710 31307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:11.747267 31307 catalog_manager.cc:1347] Generated new cluster ID: 94a7ea63675846419b653abf3c562898
I20260620 12:14:11.747368 31307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:11.767863 31307 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:11.769618 31307 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:11.783262 31307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: Generated new TSK 0
I20260620 12:14:11.784323 31307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab.
Entry for principal oryx/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab.
I20260620 12:14:11.818794 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:34705
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab
--principal=oryx/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:11.957330 31319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:11.957726 31319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:11.957780 31319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:11.962282 31319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:11.962374 31319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:11.962442 31319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:11.968668 31319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab
--principal=oryx/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:34705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:11.970299 31319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:11.971694 31319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:14:11.983978 31319 server_base.cc:1034] running on GCE node
W20260620 12:14:11.983876 31324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:11.983870 31327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:11.983870 31325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:11.984833 31319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:11.985777 31319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:11.987011 31319 hybrid_clock.cc:648] HybridClock initialized: now 1781957651986961 us; error 71 us; skew 500 ppm
Jun 20 12:14:11 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:11.990563 31319 init.cc:377] Logged in from keytab as oryx/127.24.141.1@KRBTEST.COM (short username oryx)
I20260620 12:14:11.991843 31319 webserver.cc:460] Webserver started at http://127.24.141.1:41593/ using document root <none> and password file <none>
I20260620 12:14:11.992604 31319 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:11.992686 31319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:11.992946 31319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:11.995790 31319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "1d53916f670f446b86f725d97b7fbb3c"
format_stamp: "Formatted at 2026-06-20 12:14:11 on dist-test-slave-rh5v"
server_key: "c617fb70e8364c027c65137614c961dd"
server_key_iv: "96e435a70d371b71665d3977b3d481eb"
server_key_version: "encryptionkey@0"
I20260620 12:14:11.996520 31319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "1d53916f670f446b86f725d97b7fbb3c"
format_stamp: "Formatted at 2026-06-20 12:14:11 on dist-test-slave-rh5v"
server_key: "c617fb70e8364c027c65137614c961dd"
server_key_iv: "96e435a70d371b71665d3977b3d481eb"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.002249 31319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:14:12.005911 31334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.007838 31319 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:12.008030 31319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "1d53916f670f446b86f725d97b7fbb3c"
format_stamp: "Formatted at 2026-06-20 12:14:11 on dist-test-slave-rh5v"
server_key: "c617fb70e8364c027c65137614c961dd"
server_key_iv: "96e435a70d371b71665d3977b3d481eb"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.008177 31319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:12.034828 31319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:12.038923 31319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:12.039172 31319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:12.039921 31319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:12.040978 31319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:12.041033 31319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.041124 31319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:12.041153 31319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.052567 31319 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:39599
I20260620 12:14:12.052701 31447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:39599 every 8 connection(s)
I20260620 12:14:12.053920 31319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:12.058676 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 31319
I20260620 12:14:12.058836 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:12.059180 25140 external_mini_cluster.cc:1428] Setting key ec3dd15ac21c6628564f395c3ee34bf7
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.2@KRBTEST.COM" created.
I20260620 12:14:12.073192 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.056620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:57621 (local address 127.24.141.62:34705)
0620 12:14:12.056819 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.056824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.058365 (+ 1541us) server_negotiation.cc:407] Connection header received
0620 12:14:12.059791 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.059796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.059879 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.059988 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.061761 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.062414 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.064175 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.064551 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.064721 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.067889 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.067895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.067899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.067939 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.070048 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.071109 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.071112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.071114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.071206 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.071717 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.071721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.071723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.071978 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.072075 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.072759 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.072965 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:14:12.074354 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.057175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:57621)
0620 12:14:12.058151 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.058218 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.059461 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.060255 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.060264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.060712 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.061534 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.061558 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.062587 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.062591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.063981 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.063999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.064893 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.064897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.065134 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.065825 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.065846 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.067618 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.070250 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.070259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.070282 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.070948 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.071322 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.071325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.071327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.071545 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.072128 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.072137 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.072477 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.072679 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.073123 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":161,"threads_started":1}
I20260620 12:14:12.075937 31448 heartbeater.cc:347] Connected to a master server at 127.24.141.62:34705
I20260620 12:14:12.076213 31448 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:12.076807 31448 heartbeater.cc:511] Master 127.24.141.62:34705 requested a full tablet report, sending...
I20260620 12:14:12.079093 31257 ts_manager.cc:194] Registered new tserver with Master: 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599)
I20260620 12:14:12.081029 31257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:57621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab.
Entry for principal oryx/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab.
I20260620 12:14:12.090271 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:34705
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab
--principal=oryx/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:12.224521 31453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:12.224885 31453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:12.224936 31453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:12.229357 31453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:12.229441 31453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:12.229507 31453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:12.235756 31453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab
--principal=oryx/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:34705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:12.237351 31453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:12.238730 31453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:12.251017 31459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:12.251024 31458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:12.251024 31461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:12.251703 31453 server_base.cc:1034] running on GCE node
I20260620 12:14:12.252243 31453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:12.253142 31453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:12.254343 31453 hybrid_clock.cc:648] HybridClock initialized: now 1781957652254300 us; error 68 us; skew 500 ppm
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957652, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:12.257915 31453 init.cc:377] Logged in from keytab as oryx/127.24.141.2@KRBTEST.COM (short username oryx)
I20260620 12:14:12.259235 31453 webserver.cc:460] Webserver started at http://127.24.141.2:40253/ using document root <none> and password file <none>
I20260620 12:14:12.259948 31453 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:12.260025 31453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:12.260262 31453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:12.262809 31453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "35e64331a4474036b783b1ec315708c2"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "34568c16267419b3ccc595ed629812d9"
server_key_iv: "a7999054169f2033f79d06b41e5cf4f7"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.263427 31453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "35e64331a4474036b783b1ec315708c2"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "34568c16267419b3ccc595ed629812d9"
server_key_iv: "a7999054169f2033f79d06b41e5cf4f7"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.268386 31453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:14:12.271796 31468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.273571 31453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:14:12.273736 31453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "35e64331a4474036b783b1ec315708c2"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "34568c16267419b3ccc595ed629812d9"
server_key_iv: "a7999054169f2033f79d06b41e5cf4f7"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.273937 31453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:12.298684 31453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:12.302549 31453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:12.302742 31453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:12.303444 31453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:12.304477 31453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:12.304529 31453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.304575 31453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:12.304598 31453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.316179 31453 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:44783
I20260620 12:14:12.316213 31581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:44783 every 8 connection(s)
I20260620 12:14:12.317421 31453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:12.319427 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 31453
I20260620 12:14:12.319564 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:12.319953 25140 external_mini_cluster.cc:1428] Setting key 1e7ca63c0c5e3399e6efbfc748b238f3
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957652, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.3@KRBTEST.COM" created.
I20260620 12:14:12.335712 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.320110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:59523 (local address 127.24.141.62:34705)
0620 12:14:12.320273 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.320278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.321663 (+ 1385us) server_negotiation.cc:407] Connection header received
0620 12:14:12.323334 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.323338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.323414 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.323498 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.325035 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.325640 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.326931 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.327166 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.327261 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.330543 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.330547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.330550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.330587 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.332039 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.332963 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.332967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.332969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.333120 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.333614 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.333619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.333620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.334111 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.334223 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.335241 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.335474 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:14:12.336822 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.320697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:59523)
0620 12:14:12.321466 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.321536 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.323054 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.323748 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.323757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.324232 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.324855 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.324871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.325783 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.325786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.326783 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.326796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.327346 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.327351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.327529 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.328183 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.328203 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.330163 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.332222 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.332230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.332251 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.332784 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.333234 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.333237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.333239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.333460 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.334377 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.334388 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.334855 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.335154 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.335553 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":177,"threads_started":1}
I20260620 12:14:12.338826 31582 heartbeater.cc:347] Connected to a master server at 127.24.141.62:34705
I20260620 12:14:12.339162 31582 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:12.339800 31582 heartbeater.cc:511] Master 127.24.141.62:34705 requested a full tablet report, sending...
I20260620 12:14:12.341398 31257 ts_manager.cc:194] Registered new tserver with Master: 35e64331a4474036b783b1ec315708c2 (127.24.141.2:44783)
I20260620 12:14:12.342511 31257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:59523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab.
Entry for principal oryx/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab.
I20260620 12:14:12.353260 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:34705
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab
--principal=oryx/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:12.489719 31587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:12.490140 31587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:12.490193 31587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:12.495383 31587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:12.495512 31587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:12.495587 31587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:12.502259 31587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab
--principal=oryx/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:34705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:12.503916 31587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:12.505440 31587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:12.518155 31592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:12.518177 31593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:12.518177 31595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:12.518518 31587 server_base.cc:1034] running on GCE node
I20260620 12:14:12.519156 31587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:12.519980 31587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:12.521196 31587 hybrid_clock.cc:648] HybridClock initialized: now 1781957652521159 us; error 49 us; skew 500 ppm
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957652, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:12.524637 31587 init.cc:377] Logged in from keytab as oryx/127.24.141.3@KRBTEST.COM (short username oryx)
I20260620 12:14:12.525903 31587 webserver.cc:460] Webserver started at http://127.24.141.3:45129/ using document root <none> and password file <none>
I20260620 12:14:12.526630 31587 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:12.526713 31587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:12.526955 31587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:12.529470 31587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "4f8bbe4ed35c4b8a944802876b843431"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "49ed7a0fc74181582143b7e929c68806"
server_key_iv: "5a4ca48af016ce92d080e89a91c7a67b"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.530145 31587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "4f8bbe4ed35c4b8a944802876b843431"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "49ed7a0fc74181582143b7e929c68806"
server_key_iv: "5a4ca48af016ce92d080e89a91c7a67b"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.535223 31587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:12.538767 31602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.540673 31587 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:12.540882 31587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "4f8bbe4ed35c4b8a944802876b843431"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "49ed7a0fc74181582143b7e929c68806"
server_key_iv: "5a4ca48af016ce92d080e89a91c7a67b"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.541021 31587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:12.558996 31587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:12.562880 31587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:12.563105 31587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:12.563822 31587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:12.564832 31587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:12.564885 31587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.564947 31587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:12.564976 31587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.576103 31587 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:40285
I20260620 12:14:12.576169 31715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:40285 every 8 connection(s)
I20260620 12:14:12.577280 31587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:12.582617 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 31587
I20260620 12:14:12.582785 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:12.583170 25140 external_mini_cluster.cc:1428] Setting key 63c75025ed6bab720b699dc303eca22c
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957652, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:12.594978 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.580017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:57789 (local address 127.24.141.62:34705)
0620 12:14:12.580196 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.580200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.581422 (+ 1222us) server_negotiation.cc:407] Connection header received
0620 12:14:12.583514 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.583519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.583590 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.583677 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.585262 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.585823 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.587056 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.587293 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.587385 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.590267 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.590270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.590272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.590301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.591958 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.592820 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.592824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.592826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.592934 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.593463 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.593466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.593468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.593732 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.593836 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.594627 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.594798 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:14:12.596781 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.580520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:57789)
0620 12:14:12.581219 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.581286 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.583108 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.583991 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.584002 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.584468 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.585039 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.585100 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.585963 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.585967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.586926 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.586938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.587451 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.587456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.587630 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.588254 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.588274 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.590046 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.592203 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.592212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.592237 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.592629 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.593054 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.593100 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.593103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.593304 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.593878 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.593886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.594262 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.594512 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.594934 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":146,"threads_started":1}
I20260620 12:14:12.599675 31716 heartbeater.cc:347] Connected to a master server at 127.24.141.62:34705
I20260620 12:14:12.600090 31716 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:12.600931 31716 heartbeater.cc:511] Master 127.24.141.62:34705 requested a full tablet report, sending...
I20260620 12:14:12.602551 31256 ts_manager.cc:194] Registered new tserver with Master: 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3:40285)
I20260620 12:14:12.603271 31256 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:57789
I20260620 12:14:12.612042 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.141.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.61.keytab.
Entry for principal oryx/127.24.141.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.61.keytab.
I20260620 12:14:12.642632 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:37555
--webserver_interface=127.24.141.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.61.keytab
--principal=oryx/127.24.141.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:34705,127.24.141.61:37555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:12.774988 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:12.775377 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:12.775425 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:12.779911 31721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:12.779999 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:12.780018 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:12.780037 31721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:12.780052 31721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:12.786489 31721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:34705,127.24.141.61:37555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.61.keytab
--principal=oryx/127.24.141.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:37555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.141.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
W20260620 12:14:12.786774 31721 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.141.62:34705,127.24.141.61:37555'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 12:14:12.787942 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:12.789232 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:12.799165 31729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:12.799125 31727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:12.799125 31726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:12.799336 31721 server_base.cc:1034] running on GCE node
I20260620 12:14:12.800200 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:12.801753 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:12.802942 31721 hybrid_clock.cc:648] HybridClock initialized: now 1781957652802913 us; error 50 us; skew 500 ppm
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957652, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:12.806526 31721 init.cc:377] Logged in from keytab as oryx/127.24.141.61@KRBTEST.COM (short username oryx)
I20260620 12:14:12.808054 31721 webserver.cc:460] Webserver started at http://127.24.141.61:36161/ using document root <none> and password file <none>
I20260620 12:14:12.808764 31721 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:12.808844 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:12.809054 31721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:12.811625 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/instance:
uuid: "8c59ee4a783841f7b5ed08b90271dc62"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "bd1e5af21116b9fce4d63dd4abcceeef"
server_key_iv: "62f80d8dd64a61d8b0a542da6311a260"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.812297 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal/instance:
uuid: "8c59ee4a783841f7b5ed08b90271dc62"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "bd1e5af21116b9fce4d63dd4abcceeef"
server_key_iv: "62f80d8dd64a61d8b0a542da6311a260"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.817267 31721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:14:12.821249 31736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.823264 31721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:12.823491 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
uuid: "8c59ee4a783841f7b5ed08b90271dc62"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "bd1e5af21116b9fce4d63dd4abcceeef"
server_key_iv: "62f80d8dd64a61d8b0a542da6311a260"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.823659 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:12.848915 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:12.852943 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:12.853209 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:12.862413 31721 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.61:37555
I20260620 12:14:12.862737 31788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.61:37555 every 8 connection(s)
I20260620 12:14:12.863735 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
I20260620 12:14:12.867964 31789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:12.872123 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 31721
I20260620 12:14:12.872895 31789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957652, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.61@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.141.61@KRBTEST.COM
I20260620 12:14:12.888137 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.875451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.24.141.62:34705)
0620 12:14:12.875650 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.875656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.876126 (+ 470us) server_negotiation.cc:407] Connection header received
0620 12:14:12.876660 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.876665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.876741 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.876867 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.878055 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.878630 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.879879 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.880129 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.880225 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.883193 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.883199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.883203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.883247 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.885363 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.886101 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.886104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.886106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.886205 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.886689 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.886693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.886695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.887002 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.887115 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.887809 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.887963 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:14:12.889390 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.874483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.24.141.61:37555)
0620 12:14:12.875047 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.875065 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.875109 (+ 44us) server_negotiation.cc:407] Connection header received
0620 12:14:12.876221 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.876251 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.876560 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.876755 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.877706 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.878844 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.879791 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.880257 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.880491 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.883589 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.883597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.883610 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.883634 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.885872 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.886485 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.886493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.886494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.886600 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.886915 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.886918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.886919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.887508 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.887711 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.888003 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.888351 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":385,"spinlock_wait_cycles":3840,"thread_start_us":184,"threads_started":1}
I20260620 12:14:12.889406 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.875387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:42806)
0620 12:14:12.875916 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.875987 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.876401 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.876955 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.876961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.877484 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.877849 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.877863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.878782 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.878786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.879722 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.879732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.880304 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.880310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.880506 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.881256 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.881274 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.882917 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.885543 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.885552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.885564 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.885970 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.886306 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.886309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.886311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.886552 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.887124 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.887131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.887527 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.887740 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.888172 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":61,"thread_start_us":182,"threads_started":1}
I20260620 12:14:12.891347 31789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957652, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.61@KRBTEST.COM for oryx/127.24.141.61@KRBTEST.COM
I20260620 12:14:12.900151 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.891936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.0.0.1:54668)
0620 12:14:12.892054 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.892072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.892148 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.892706 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.892708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.892867 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.892989 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.892994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.893826 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.893829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.894419 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.894426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.894928 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.894931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.895015 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.895502 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.895513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.896899 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.898566 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.898568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.898570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.898778 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.899101 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.899103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.899105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.899177 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.899629 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.899631 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.899732 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.899836 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.899983 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 12:14:12.900274 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.892307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.24.141.61:37555)
0620 12:14:12.892473 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.892477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.892492 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:14:12.892530 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.892532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.892581 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.892677 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.893135 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.893706 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.894544 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.894810 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.894906 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.897094 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.897098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.897102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.897139 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.898431 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.898911 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.898915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.898916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.899010 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.899285 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.899288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.899290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.899503 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.899601 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.899941 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.900080 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:14:12.903805 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Bootstrap starting.
I20260620 12:14:12.906581 31789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:12.907426 31789 log.cc:824] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:12.910105 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: No bootstrap required, opened a new log
I20260620 12:14:12.913658 31789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } }
I20260620 12:14:12.913949 31789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:12.913986 31789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c59ee4a783841f7b5ed08b90271dc62, State: Initialized, Role: FOLLOWER
I20260620 12:14:12.914561 31789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } }
I20260620 12:14:12.915386 31795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } } }
I20260620 12:14:12.915606 31795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:14:12.916705 31789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 12:14:12.923081 31806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:14:12.923236 31806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:14:12.925983 31721 master_runner.cc:428] Detected that this master 8c59ee4a783841f7b5ed08b90271dc62 is joining an existing cluster
I20260620 12:14:12.926074 31721 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.141.61:37555
I20260620 12:14:12.936894 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.929092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.24.141.61:37555)
0620 12:14:12.929206 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.929209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.929313 (+ 104us) server_negotiation.cc:407] Connection header received
0620 12:14:12.929445 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.929448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.929486 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.929551 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.930023 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.930537 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.931797 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.932088 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.932191 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.933610 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.933613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.933616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.933636 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.935181 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.935631 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.935635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.935637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.935717 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.935967 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.935970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.935972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.936166 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.936250 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.936495 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.936754 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":46}
I20260620 12:14:12.936911 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.928656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.0.0.1:54682)
0620 12:14:12.929229 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.929243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.929363 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.929578 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.929581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.929806 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.929916 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.929922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.930664 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.930667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.931662 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.931672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.932221 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.932225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.932315 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.932880 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.932894 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.933474 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.935295 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.935298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.935300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.935545 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.935812 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.935815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.935817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.935894 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.936252 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.936255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.936335 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.936441 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.936682 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"spinlock_wait_cycles":12160,"thread_start_us":120,"threads_started":1}
I20260620 12:14:12.938247 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.928756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.24.141.62:34705)
0620 12:14:12.928895 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.928900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.929455 (+ 555us) server_negotiation.cc:407] Connection header received
0620 12:14:12.929670 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.929674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.929741 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.929798 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.930576 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.931124 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.932117 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.932486 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.932609 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.933644 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.933648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.933651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.933679 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.936196 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.936817 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.936821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.936823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.936910 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.937236 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.937240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.937241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.937495 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.937567 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.937908 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.938052 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:14:12.938803 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.928808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:42822)
0620 12:14:12.929322 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.929338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.929545 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.929973 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.929976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.930264 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.930404 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.930425 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.931236 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.931239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.931966 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.931974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.932609 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.932632 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.932708 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.933120 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.933134 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.933496 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.936338 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.936342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.936345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.936705 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.937021 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.937024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.937026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.937138 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.937637 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.937640 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.937734 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.938434 (+ 700us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.938608 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":179,"threads_started":1}
Jun 20 12:14:12 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.62@KRBTEST.COM for oryx/127.24.141.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.141.60@KRBTEST.COM; defaulting to no policy
I20260620 12:14:12.957532 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.944875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.24.141.61:37555)
0620 12:14:12.945015 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:12.945020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:12.946481 (+ 1461us) server_negotiation.cc:407] Connection header received
0620 12:14:12.946739 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:12.946743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:12.946823 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:12.946948 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:12.947738 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.948581 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.949735 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.950036 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.950146 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.953121 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:12.953125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:12.953128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:12.953161 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:12.955057 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.955818 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.955822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.955823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.955906 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:12.956240 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:12.956243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:12.956245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:12.956520 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:12.956622 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:12.957149 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 12:14:12.957304 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:14:12.957638 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:12.944842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.0.0.1:54686)
0620 12:14:12.946305 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:12.946374 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:12.946564 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:12.947018 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:12.947025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:12.947462 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:12.947608 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.947615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.948732 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.948736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.949593 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:12.949606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:12.950170 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:12.950173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:12.950281 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:12.950925 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:12.950941 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:12.952888 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:12.955229 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.955244 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.955252 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.955672 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.956000 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:12.956004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:12.956005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:12.956106 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:12.956643 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:12.956652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:12.956836 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 12:14:12.957019 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 12:14:12.957377 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.60@KRBTEST.COM" created.
I20260620 12:14:12.959966 31256 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.141.61:37555: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62"
member_type: NON_VOTER
last_known_addr {
host: "127.24.141.61"
port: 37555
}
attrs {
promote: true
}
}
dest_uuid: "b49e4418976f48feace83ae97c424b7c"
cas_config_opid_index: -1
I20260620 12:14:12.961333 31256 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } }
I20260620 12:14:12.963907 31819 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8c59ee4a783841f7b5ed08b90271dc62 (127.24.141.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } }
I20260620 12:14:12.964877 31763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:12.966758 31819 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.141.61:37555
I20260620 12:14:12.967113 31763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Refusing update from remote peer b49e4418976f48feace83ae97c424b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 12:14:12.967737 31721 master.cc:537] Master@127.24.141.61:37555 shutting down...
I20260620 12:14:12.968384 31819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } }
I20260620 12:14:12.968674 31819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:12.968986 31819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:14:12.972690 31241 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 12:14:12.974071 31721 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 12:14:12.974267 31241 consensus_peers.cc:487] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c -> Peer 8c59ee4a783841f7b5ed08b90271dc62 (127.24.141.61:37555): Couldn't send request to peer 8c59ee4a783841f7b5ed08b90271dc62. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 12:14:12.974496 31721 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:12.974624 31721 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.60.keytab.
Entry for principal oryx/127.24.141.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.60.keytab.
I20260620 12:14:12.981036 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:41893
--webserver_interface=127.24.141.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.60.keytab
--principal=oryx/127.24.141.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:14:12.991782 31721 master.cc:552] Master@127.24.141.61:37555 shutdown complete.
I20260620 12:14:12.991873 31721 master_runner.cc:305] Clearing existing system tablet
I20260620 12:14:12.992151 31721 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:12.992215 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:12.993835 31721 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 12:14:12.994904 31826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:12.995648 31721 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:12.995820 31721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:12.995870 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
uuid: "8c59ee4a783841f7b5ed08b90271dc62"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "bd1e5af21116b9fce4d63dd4abcceeef"
server_key_iv: "62f80d8dd64a61d8b0a542da6311a260"
server_key_version: "encryptionkey@0"
I20260620 12:14:12.998476 31721 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:13.001392 31721 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 12:14:13.001523 31721 log.cc:1196] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 12:14:13.001819 31721 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Deleting consensus metadata
I20260620 12:14:13.001999 31721 master_runner.cc:315] Copying system tablet from 127.24.141.62:34705
I20260620 12:14:13.003543 31721 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: tablet copy: Beginning tablet copy session from remote peer at address 127.24.141.62:34705
I20260620 12:14:13.015343 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.004552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.24.141.62:34705)
0620 12:14:13.004713 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.004717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.005087 (+ 370us) server_negotiation.cc:407] Connection header received
0620 12:14:13.005452 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.005457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.005521 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.005579 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.006405 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.007283 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.008638 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.009000 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.009210 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.010778 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.010783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.010786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.010823 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.012959 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.013671 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.013678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.013680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.013871 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.014211 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.014214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.014216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.014498 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.014613 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.014937 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.015114 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:14:13.015344 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.004435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:42832)
0620 12:14:13.004907 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.004925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.005233 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.005770 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.005773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.006054 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.006210 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.006218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.007467 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.007472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.008447 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.008466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.009220 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.009224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.009363 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.010064 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.010080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.010557 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.013165 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.013170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.013173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.013500 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.014004 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.014006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.014009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.014108 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.014602 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.014606 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.014747 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.014869 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.015082 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":127,"threads_started":1}
I20260620 12:14:13.016624 31277 tablet_copy_service.cc:140] P b49e4418976f48feace83ae97c424b7c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c59ee4a783841f7b5ed08b90271dc62 ({username='oryx', principal='oryx/127.24.141.61@KRBTEST.COM'} at 127.0.0.1:42832)
I20260620 12:14:13.016825 31277 tablet_copy_service.cc:161] P b49e4418976f48feace83ae97c424b7c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c59ee4a783841f7b5ed08b90271dc62 at {username='oryx', principal='oryx/127.24.141.61@KRBTEST.COM'} at 127.0.0.1:42832: session id = 8c59ee4a783841f7b5ed08b90271dc62-00000000000000000000000000000000
I20260620 12:14:13.018949 31277 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 12:14:13.021397 31721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 12:14:13.024624 31721 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: tablet copy: Starting download of 0 data blocks...
I20260620 12:14:13.024896 31721 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: tablet copy: Starting download of 1 WAL segments...
I20260620 12:14:13.027696 31721 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 12:14:13.030581 31277 tablet_copy_service.cc:342] P b49e4418976f48feace83ae97c424b7c: Request end of tablet copy session 8c59ee4a783841f7b5ed08b90271dc62-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.141.61@KRBTEST.COM'} at 127.0.0.1:42832
I20260620 12:14:13.030788 31277 tablet_copy_service.cc:433] P b49e4418976f48feace83ae97c424b7c: ending tablet copy session 8c59ee4a783841f7b5ed08b90271dc62-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c59ee4a783841f7b5ed08b90271dc62
I20260620 12:14:13.041445 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:13.043625 31833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.043715 31834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.044278 31836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:13.044539 31721 server_base.cc:1034] running on GCE node
I20260620 12:14:13.044795 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:13.045222 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:13.046527 31721 hybrid_clock.cc:648] HybridClock initialized: now 1781957653046513 us; error 34 us; skew 500 ppm
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:13.048400 31721 init.cc:377] Logged in from keytab as oryx/127.24.141.61@KRBTEST.COM (short username oryx)
I20260620 12:14:13.049124 31721 webserver.cc:460] Webserver started at http://127.24.141.61:46605/ using document root <none> and password file <none>
I20260620 12:14:13.049371 31721 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:13.049474 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:13.051050 31721 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:13.052259 31843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.053445 31721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:14:13.053539 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
uuid: "8c59ee4a783841f7b5ed08b90271dc62"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "bd1e5af21116b9fce4d63dd4abcceeef"
server_key_iv: "62f80d8dd64a61d8b0a542da6311a260"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.053642 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:13.084285 31448 heartbeater.cc:503] Master 127.24.141.62:34705 was elected leader, sending a full tablet report...
I20260620 12:14:13.094885 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:13.097940 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:13.103665 31721 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.61:37555
I20260620 12:14:13.104285 31895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.61:37555 every 8 connection(s)
I20260620 12:14:13.104575 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
I20260620 12:14:13.106309 31896 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:14:13.107298 31896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Bootstrap starting.
I20260620 12:14:13.117867 31896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:14:13.118175 31896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Bootstrap complete.
I20260620 12:14:13.118736 31896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } }
I20260620 12:14:13.118856 31896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c59ee4a783841f7b5ed08b90271dc62, State: Initialized, Role: LEARNER
I20260620 12:14:13.118953 31896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } }
I20260620 12:14:13.119412 31896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:13.119398 31899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } }
I20260620 12:14:13.119614 31899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:14:13.123703 31910 catalog_manager.cc:1259] Loaded cluster ID: 94a7ea63675846419b653abf3c562898
I20260620 12:14:13.123811 31910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: loading cluster ID for follower catalog manager: success
I20260620 12:14:13.126307 31910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: acquiring CA information for follower catalog manager: success
I20260620 12:14:13.128170 31910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 12:14:13.138644 31824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:13.138995 31824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:13.139042 31824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:13.143375 31824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:13.143450 31824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:13.143469 31824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:13.143486 31824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:13.143500 31824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:13.149552 31824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.60.keytab
--principal=oryx/127.24.141.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:41893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.141.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:13.150910 31824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:13.152136 31824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:13.161980 31915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.162057 31916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.161983 31918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:13.162634 31824 server_base.cc:1034] running on GCE node
I20260620 12:14:13.163443 31824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:13.164829 31824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:13.166054 31824 hybrid_clock.cc:648] HybridClock initialized: now 1781957653166032 us; error 68 us; skew 500 ppm
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:13.169780 31824 init.cc:377] Logged in from keytab as oryx/127.24.141.60@KRBTEST.COM (short username oryx)
I20260620 12:14:13.171259 31824 webserver.cc:460] Webserver started at http://127.24.141.60:45941/ using document root <none> and password file <none>
I20260620 12:14:13.172027 31824 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:13.172106 31824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:13.172359 31824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:13.175026 31824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/instance:
uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
format_stamp: "Formatted at 2026-06-20 12:14:13 on dist-test-slave-rh5v"
server_key: "a3380464da407dbf8e4d9839f92110a0"
server_key_iv: "d73ee475d3ac0feb417362876555c3ca"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.175685 31824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal/instance:
uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
format_stamp: "Formatted at 2026-06-20 12:14:13 on dist-test-slave-rh5v"
server_key: "a3380464da407dbf8e4d9839f92110a0"
server_key_iv: "d73ee475d3ac0feb417362876555c3ca"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.180727 31824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:14:13.184018 31925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.186126 31824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:14:13.186322 31824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
format_stamp: "Formatted at 2026-06-20 12:14:13 on dist-test-slave-rh5v"
server_key: "a3380464da407dbf8e4d9839f92110a0"
server_key_iv: "d73ee475d3ac0feb417362876555c3ca"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.186632 31824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:13.202163 31824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:13.205791 31824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:13.205998 31824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:13.213760 31824 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.60:41893
I20260620 12:14:13.214076 31977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.60:41893 every 8 connection(s)
I20260620 12:14:13.214991 31824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
I20260620 12:14:13.218779 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:13.220038 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 31824
I20260620 12:14:13.224081 31978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.141.60@KRBTEST.COM
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.60@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:13.237389 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.226663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.24.141.62:34705)
0620 12:14:13.226821 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.226825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.227126 (+ 301us) server_negotiation.cc:407] Connection header received
0620 12:14:13.227311 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.227314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.227379 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.227474 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.228239 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.228767 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.229949 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.230316 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.230441 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.233340 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.233344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.233347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.233377 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.235239 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.235785 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.235788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.235789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.235895 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.236222 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.236226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.236227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.236480 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.236587 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.237037 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.237208 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:14:13.238758 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.222074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.24.141.60:41893)
0620 12:14:13.222757 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.222777 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.222829 (+ 52us) server_negotiation.cc:407] Connection header received
0620 12:14:13.223735 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.223759 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.224231 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.224661 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.225758 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.227175 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.228236 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.228795 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.228986 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.232160 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.232169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.232182 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.232220 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.234391 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.234912 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.234918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.234925 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.235071 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.235339 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.235343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.235345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.236063 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.236295 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.237041 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.237403 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":452,"thread_start_us":195,"threads_started":1}
I20260620 12:14:13.238755 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.226376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:42848)
0620 12:14:13.226946 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.227008 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.227191 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.227533 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.227539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.227959 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.228101 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.228107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.228897 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.228900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.229817 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.229828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.230595 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.230604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.230712 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.231381 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.231404 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.233133 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.235373 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.235382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.235390 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.235671 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.235989 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.235991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.235993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.236076 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.236600 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.236606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.236746 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.236932 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.237433 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":98,"thread_start_us":185,"threads_started":1}
I20260620 12:14:13.240435 31978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.60@KRBTEST.COM for oryx/127.24.141.61@KRBTEST.COM
I20260620 12:14:13.251500 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.241055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.0.0.1:54700)
0620 12:14:13.241322 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.241338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.241448 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.241890 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.241892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.242076 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.242218 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.242224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.243295 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.243299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.243980 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.243991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.244607 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.244611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.244747 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.245489 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.245502 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.247137 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.249626 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.249631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.249633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.249893 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.250288 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.250291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.250293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.250386 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.250914 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.250916 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.251048 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.251156 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.251319 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 12:14:13.251608 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.241239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.24.141.61:37555)
0620 12:14:13.241619 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.241623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.241641 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:14:13.241689 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.241700 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.241745 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.241869 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.242365 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.243121 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.244147 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.244439 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.244555 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.247399 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.247403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.247405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.247444 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.249432 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.250042 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.250046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.250047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.250171 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.250534 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.250538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.250539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.250785 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.250885 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.251289 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.251458 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":140,"threads_started":1}
I20260620 12:14:13.252687 31978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.60@KRBTEST.COM for oryx/127.24.141.60@KRBTEST.COM
I20260620 12:14:13.261814 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.253436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.0.0.1:49432)
0620 12:14:13.253631 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.253648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.253794 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.254014 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.254016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.254225 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.254310 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.254316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.255482 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.255486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.256017 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.256025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.256484 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.256488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.256555 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.257077 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.257095 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.258527 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.260202 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.260205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.260207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.260428 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.260731 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.260734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.260735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.260823 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.261256 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.261259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.261377 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.261484 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.261642 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 12:14:13.261873 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.253510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.24.141.60:41893)
0620 12:14:13.253631 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.253636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.253724 (+ 88us) server_negotiation.cc:407] Connection header received
0620 12:14:13.253881 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.253884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.253924 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.253990 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.254427 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.255350 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.256133 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.256375 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.256465 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.258724 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.258729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.258731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.258758 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.260062 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.260520 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.260522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.260524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.260620 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.260917 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.260919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.260920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.261158 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.261242 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.261535 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.261717 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":49}
I20260620 12:14:13.265105 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Bootstrap starting.
I20260620 12:14:13.267758 31978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:13.268570 31978 log.cc:824] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:13.271951 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: No bootstrap required, opened a new log
I20260620 12:14:13.275450 31978 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } }
I20260620 12:14:13.275760 31978 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:13.275830 31978 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ec80927b884b3d8a46fd7d9601f85e, State: Initialized, Role: FOLLOWER
I20260620 12:14:13.276672 31978 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } }
I20260620 12:14:13.277456 31984 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } } }
I20260620 12:14:13.277668 31984 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:14:13.278373 31978 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 12:14:13.284638 31995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:14:13.284755 31995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:14:13.288944 31824 master_runner.cc:428] Detected that this master e4ec80927b884b3d8a46fd7d9601f85e is joining an existing cluster
I20260620 12:14:13.289108 31824 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.141.60:41893
I20260620 12:14:13.293545 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 31319
I20260620 12:14:13.302125 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:39599
--local_ip_for_outbound_sockets=127.24.141.1
--tserver_master_addrs=127.24.141.62:34705
--webserver_port=41593
--webserver_interface=127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab
--principal=oryx/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:14:13.303328 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.292934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.0.0.1:49442)
0620 12:14:13.295012 (+ 2078us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.295025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.295138 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.295408 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.295411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.295594 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.295710 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.295717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.296819 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.296822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.297789 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.297797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.298643 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.298645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.298704 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.299082 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.299093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.299472 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.301114 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.301117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.301120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.301294 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.301533 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.301537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.301539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.301611 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.302589 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.302593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.302695 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.302833 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.303046 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2002,"mutex_wait_us":42,"thread_start_us":139,"threads_started":1}
I20260620 12:14:13.303341 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.292914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.24.141.60:41893)
0620 12:14:13.293179 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.293183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.295090 (+ 1907us) server_negotiation.cc:407] Connection header received
0620 12:14:13.295220 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.295223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.295261 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.295302 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.295832 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.296702 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.297888 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.298485 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.298540 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.299593 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.299597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.299600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.299622 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.301001 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.301375 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.301379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.301381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.301455 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.301770 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.301777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.301779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.302110 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.302260 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.302932 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.303133 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":188}
I20260620 12:14:13.331315 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.292206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:42858)
0620 12:14:13.293140 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.293159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.293359 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.293841 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.293843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.294110 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.294272 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.294278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.295468 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.295472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.296640 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.296658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.297589 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.297592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.297790 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.298407 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.298422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.325619 (+ 27197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.328034 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.328038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.328041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.328299 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.329038 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.329043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.329045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.329244 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.330512 (+ 1268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.330516 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.330679 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.330801 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.331053 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":68,"thread_start_us":169,"threads_started":1}
I20260620 12:14:13.331475 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.292400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.24.141.62:34705)
0620 12:14:13.292777 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.292781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.293258 (+ 477us) server_negotiation.cc:407] Connection header received
0620 12:14:13.293551 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.293555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.293617 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.293671 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.294430 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.295277 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.296818 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.297219 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.297319 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.325966 (+ 28647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.325972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.325976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.326021 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.327792 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.328473 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.328479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.328482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.328709 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.329737 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.329742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.329744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.330324 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.330450 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.331126 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.331275 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260620 12:14:13.333832 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.292206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.24.141.61:37555)
0620 12:14:13.292390 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.292395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.293424 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 12:14:13.293588 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.293591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.293645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.293694 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.294909 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.295743 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.297423 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.297789 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.297870 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.328911 (+ 31041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.328928 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.328932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.328976 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.330746 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.332308 (+ 1562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.332312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.332314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.332409 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.332694 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.332697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.332699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.332961 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.333044 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.333473 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.333645 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 12:14:13.334486 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.292503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.0.0.1:54704)
0620 12:14:13.293339 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.293350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.293488 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.294519 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.294521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.294685 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.294785 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.294789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.295858 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.295860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.297284 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.297294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.298041 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.298044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.298129 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.326450 (+ 28321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.326496 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.328690 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.330891 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.330895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.330898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.332180 (+ 1282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.332514 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.332517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.332519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.332602 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.333154 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.333157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.333269 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.334082 (+ 813us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.334300 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":19,"thread_start_us":120,"threads_started":1}
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957651, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.62@KRBTEST.COM for oryx/127.24.141.60@KRBTEST.COM
I20260620 12:14:13.345505 31582 heartbeater.cc:503] Master 127.24.141.62:34705 was elected leader, sending a full tablet report...
I20260620 12:14:13.347714 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.336592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.0.0.1:49448)
0620 12:14:13.336737 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.336754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.336842 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.337365 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.337368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.337593 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.337736 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.337743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.338810 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.338813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.339756 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.339767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.340441 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.340445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.340552 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.341266 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.341280 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.342955 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.345416 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.345421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.345424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.345956 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.346345 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.346349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.346351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.346459 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.347007 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.347011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.347138 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.347273 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.347488 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 12:14:13.347733 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.336823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.24.141.60:41893)
0620 12:14:13.336997 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.337002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.337023 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:14:13.337153 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.337157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.337232 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.337341 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.337873 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.338674 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.339907 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.340292 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.340440 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.343162 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.343168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.343172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.343217 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.345203 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.346108 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.346112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.346114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.346221 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.346588 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.346591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.346593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.346852 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.346956 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.347383 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.347547 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:14:13.350054 31256 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.141.60:41893: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
member_type: NON_VOTER
last_known_addr {
host: "127.24.141.60"
port: 41893
}
attrs {
promote: true
}
}
dest_uuid: "b49e4418976f48feace83ae97c424b7c"
cas_config_opid_index: 5
I20260620 12:14:13.350853 31256 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } }
I20260620 12:14:13.352756 31821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.353984 31952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:13.355136 31821 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.141.60:41893
I20260620 12:14:13.355408 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.355532 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:13.355916 31952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Refusing update from remote peer b49e4418976f48feace83ae97c424b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 12:14:13.356815 31824 master.cc:537] Master@127.24.141.60:41893 shutting down...
I20260620 12:14:13.359694 31821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:14:13.363315 31241 consensus_peers.cc:487] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c -> Peer e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60:41893): Couldn't send request to peer e4ec80927b884b3d8a46fd7d9601f85e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 12:14:13.364099 31824 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:13.364512 31824 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:13.364630 31824 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: stopping tablet replica
I20260620 12:14:13.370138 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.352893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.0.0.1:54710)
0620 12:14:13.356280 (+ 3387us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.356305 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.356445 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.364124 (+ 7679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.364128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.364152 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:13.364402 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.364411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.365726 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.365730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.367035 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.367050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.369157 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.369161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.369299 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.369385 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.369826 (+ 441us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.369951 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3288}
I20260620 12:14:13.370718 31824 master.cc:552] Master@127.24.141.60:41893 shutdown complete.
I20260620 12:14:13.370798 31824 master_runner.cc:305] Clearing existing system tablet
I20260620 12:14:13.370782 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.353192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.24.141.61:37555)
0620 12:14:13.356954 (+ 3762us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.356959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.356978 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:14:13.363772 (+ 6794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.363777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.363866 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.363966 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:13.364553 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.365421 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.367227 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.368582 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.368711 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.368947 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.370508 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.370637 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3651}
I20260620 12:14:13.371085 31824 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:13.371280 31824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:13.372138 31870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 LEARNER]: Refusing update from remote peer b49e4418976f48feace83ae97c424b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 12:14:13.372776 31821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 12:14:13.375576 31870 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.377802 31824 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260620 12:14:13.378819 31899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49e4418976f48feace83ae97c424b7c. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.378975 31899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:14:13.379077 31899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: NON_VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: true } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.379177 31899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:14:13.379240 32008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.379518 31824 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:13.379679 31824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:13.379719 31824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
format_stamp: "Formatted at 2026-06-20 12:14:13 on dist-test-slave-rh5v"
server_key: "a3380464da407dbf8e4d9839f92110a0"
server_key_iv: "d73ee475d3ac0feb417362876555c3ca"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.380306 31821 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: attempting to promote NON_VOTER 8c59ee4a783841f7b5ed08b90271dc62 to VOTER
I20260620 12:14:13.380952 31821 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } }
I20260620 12:14:13.382758 31824 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:13.383000 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.382622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.0.0.1:49452)
0620 12:14:13.382798 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.382892 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:41893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 12:14:13.383673 31869 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 LEARNER]: Refusing update from remote peer b49e4418976f48feace83ae97c424b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 12:14:13.384042 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.383794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.0.0.1:49466)
0620 12:14:13.383923 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.383971 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:41893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 12:14:13.384282 31241 consensus_peers.cc:487] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c -> Peer e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60:41893): Couldn't send request to peer e4ec80927b884b3d8a46fd7d9601f85e. Status: Network error: Client connection negotiation failed: client connection to 127.24.141.60:41893: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 12:14:13.384573 31821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 12:14:13.386756 31821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8c59ee4a783841f7b5ed08b90271dc62 (127.24.141.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.386953 31824 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 12:14:13.387257 31824 log.cc:1196] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 12:14:13.387657 31824 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Deleting consensus metadata
I20260620 12:14:13.387892 31824 master_runner.cc:315] Copying system tablet from 127.24.141.62:34705
I20260620 12:14:13.389426 31824 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.141.62:34705
I20260620 12:14:13.389704 31869 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8c59ee4a783841f7b5ed08b90271dc62 (127.24.141.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.391956 31899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.392084 31899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:14:13.392458 31818 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.392613 31818 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:13.392520 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.392961 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:13.402983 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.390824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:42864)
0620 12:14:13.391165 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.391183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.391309 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.393247 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.393252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.393559 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.394050 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.394062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.395359 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.395364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.396417 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.396436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.397226 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.397231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.397374 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.398058 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.398073 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.398598 (+ 525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.400909 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.400914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.400916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.401282 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.401658 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.401662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.401664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.401753 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.402319 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.402324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.402457 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.402575 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.402772 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":78,"threads_started":1}
I20260620 12:14:13.403306 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.391522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.24.141.62:34705)
0620 12:14:13.391701 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.391705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.391722 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:14:13.391774 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.391778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.391831 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.391916 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.394239 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.395143 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.396648 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.396994 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.397193 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.398814 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.398819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.398822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.398861 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.400747 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.401418 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.401422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.401424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.401552 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.401846 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.401850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.401852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.402104 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.402245 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.402709 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.402886 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:14:13.403965 31277 tablet_copy_service.cc:140] P b49e4418976f48feace83ae97c424b7c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e4ec80927b884b3d8a46fd7d9601f85e ({username='oryx', principal='oryx/127.24.141.60@KRBTEST.COM'} at 127.0.0.1:42864)
I20260620 12:14:13.404084 31277 tablet_copy_service.cc:161] P b49e4418976f48feace83ae97c424b7c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e4ec80927b884b3d8a46fd7d9601f85e at {username='oryx', principal='oryx/127.24.141.60@KRBTEST.COM'} at 127.0.0.1:42864: session id = e4ec80927b884b3d8a46fd7d9601f85e-00000000000000000000000000000000
I20260620 12:14:13.405526 31277 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 12:14:13.407368 31824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 12:14:13.410385 31824 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: tablet copy: Starting download of 0 data blocks...
I20260620 12:14:13.410614 31824 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: tablet copy: Starting download of 1 WAL segments...
I20260620 12:14:13.412902 31824 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 12:14:13.415343 31277 tablet_copy_service.cc:342] P b49e4418976f48feace83ae97c424b7c: Request end of tablet copy session e4ec80927b884b3d8a46fd7d9601f85e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.141.60@KRBTEST.COM'} at 127.0.0.1:42864
I20260620 12:14:13.415467 31277 tablet_copy_service.cc:433] P b49e4418976f48feace83ae97c424b7c: ending tablet copy session e4ec80927b884b3d8a46fd7d9601f85e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e4ec80927b884b3d8a46fd7d9601f85e
I20260620 12:14:13.426610 31824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:13.428942 32018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.428946 32015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.429416 32016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:13.429679 31824 server_base.cc:1034] running on GCE node
I20260620 12:14:13.429893 31824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:13.430292 31824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:13.431456 31824 hybrid_clock.cc:648] HybridClock initialized: now 1781957653431430 us; error 47 us; skew 500 ppm
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:13.433449 31824 init.cc:377] Logged in from keytab as oryx/127.24.141.60@KRBTEST.COM (short username oryx)
I20260620 12:14:13.434088 31824 webserver.cc:460] Webserver started at http://127.24.141.60:39931/ using document root <none> and password file <none>
I20260620 12:14:13.434326 31824 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:13.434381 31824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:13.435957 31824 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:13.437386 32025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.437798 31824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:13.437908 31824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
format_stamp: "Formatted at 2026-06-20 12:14:13 on dist-test-slave-rh5v"
server_key: "a3380464da407dbf8e4d9839f92110a0"
server_key_iv: "d73ee475d3ac0feb417362876555c3ca"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.438010 31824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:13.459208 31824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:13.461560 31824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:13.466805 31824 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.60:41893
I20260620 12:14:13.467590 31824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
I20260620 12:14:13.468948 32077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.60:41893 every 8 connection(s)
I20260620 12:14:13.471036 32078 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:14:13.472465 32078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Bootstrap starting.
W20260620 12:14:13.476197 32004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:13.476599 32004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:13.476683 32004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:13.482707 32004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:13.482826 32004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:13.482932 32004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:13.486404 32078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:14:13.486799 32078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: Bootstrap complete.
I20260620 12:14:13.487493 32078 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } }
I20260620 12:14:13.487666 32078 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4ec80927b884b3d8a46fd7d9601f85e, State: Initialized, Role: LEARNER
I20260620 12:14:13.487792 32078 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } }
I20260620 12:14:13.488257 32084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:13.488413 32078 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:13.488423 32084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: This master's current role is: LEARNER
I20260620 12:14:13.491046 32004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab
--principal=oryx/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:39599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=41593
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:13.492481 32095 catalog_manager.cc:1259] Loaded cluster ID: 94a7ea63675846419b653abf3c562898
I20260620 12:14:13.492549 32095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: loading cluster ID for follower catalog manager: success
I20260620 12:14:13.492849 32004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:13.494129 32004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:14:13.494517 32095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: acquiring CA information for follower catalog manager: success
I20260620 12:14:13.495779 32095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 12:14:13.506327 32097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.506407 32100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.506407 32098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:13.506633 32004 server_base.cc:1034] running on GCE node
I20260620 12:14:13.507129 32004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:13.507891 32004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:13.509153 32004 hybrid_clock.cc:648] HybridClock initialized: now 1781957653509104 us; error 68 us; skew 500 ppm
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:13.513172 32004 init.cc:377] Logged in from keytab as oryx/127.24.141.1@KRBTEST.COM (short username oryx)
I20260620 12:14:13.514389 32004 webserver.cc:460] Webserver started at http://127.24.141.1:41593/ using document root <none> and password file <none>
I20260620 12:14:13.515105 32004 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:13.515192 32004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:13.520517 32004 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:13.523463 32107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.525322 32004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 12:14:13.525486 32004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "1d53916f670f446b86f725d97b7fbb3c"
format_stamp: "Formatted at 2026-06-20 12:14:11 on dist-test-slave-rh5v"
server_key: "c617fb70e8364c027c65137614c961dd"
server_key_iv: "96e435a70d371b71665d3977b3d481eb"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.526232 32004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:13.543743 32004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:13.547574 32004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:13.547766 32004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:13.549356 32004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:13.550469 32004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:13.550522 32004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.550582 32004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:13.550611 32004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.562361 32004 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:39599
I20260620 12:14:13.562402 32220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:39599 every 8 connection(s)
I20260620 12:14:13.563612 32004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:13.569620 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 32004
I20260620 12:14:13.569864 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 31453
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.61@KRBTEST.COM
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.60@KRBTEST.COM
I20260620 12:14:13.580816 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:44783
--local_ip_for_outbound_sockets=127.24.141.2
--tserver_master_addrs=127.24.141.62:34705
--webserver_port=40253
--webserver_interface=127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab
--principal=oryx/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:14:13.584261 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.565929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:42391 (local address 127.24.141.61:37555)
0620 12:14:13.566079 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.566084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.567351 (+ 1267us) server_negotiation.cc:407] Connection header received
0620 12:14:13.568786 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.568790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.568869 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.568926 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.570910 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.571747 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.573847 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.574237 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.574325 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.579222 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.579228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.579231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.579268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.581404 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.582233 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.582236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.582238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.582330 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.582757 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.582760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.582762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.583048 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.583117 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.583906 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.584066 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:14:13.584906 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.565971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:50925 (local address 127.24.141.62:34705)
0620 12:14:13.566154 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.566160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.567357 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 12:14:13.568872 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.568876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.568935 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.569013 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.571932 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.572694 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.573870 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.574212 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.574311 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.579234 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.579239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.579243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.579276 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.581804 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.582402 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.582406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.582408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.582507 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.582853 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.582857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.582859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.583108 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.583205 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.584615 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.584756 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 12:14:13.585620 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.566498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.24.141.1:42391)
0620 12:14:13.567144 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.567207 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.568500 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.569144 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.569148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.569888 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.570715 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.570727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.571933 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.571937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.573683 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.573696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.574521 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.574524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.574904 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.576094 (+ 1190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.576110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.578997 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.581604 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.581613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.581641 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.582105 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.582429 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.582432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.582434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.582636 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.583175 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.583182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.583560 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.583768 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.584212 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":210,"spinlock_wait_cycles":64512,"thread_start_us":190,"threads_started":1}
I20260620 12:14:13.585866 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.566497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:50925)
0620 12:14:13.567149 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.567208 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.568505 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.569117 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.569130 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.569858 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.570705 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.570725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.572842 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.572846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.573752 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.573759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.574520 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.574525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.574900 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.575861 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.575874 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.578790 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.581958 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.581962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.581965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.582267 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.582615 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.582618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.582632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.582733 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.583252 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.583255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.583558 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.583767 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.584300 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":232,"spinlock_wait_cycles":32896,"thread_start_us":150,"threads_started":1}
I20260620 12:14:13.587466 32223 heartbeater.cc:347] Connected to a master server at 127.24.141.61:37555
I20260620 12:14:13.587769 32223 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:13.588449 32223 heartbeater.cc:511] Master 127.24.141.61:37555 requested a full tablet report, sending...
I20260620 12:14:13.589334 32222 heartbeater.cc:347] Connected to a master server at 127.24.141.62:34705
I20260620 12:14:13.589461 32222 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:13.589721 32222 heartbeater.cc:511] Master 127.24.141.62:34705 requested a full tablet report, sending...
I20260620 12:14:13.590481 31860 ts_manager.cc:194] Registered new tserver with Master: 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599)
I20260620 12:14:13.590674 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.568021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:35577 (local address 127.24.141.60:41893)
0620 12:14:13.568340 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.568345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.568454 (+ 109us) server_negotiation.cc:407] Connection header received
0620 12:14:13.568729 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.568732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.568787 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.568871 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.571068 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.572069 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.573849 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.574236 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.574322 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.579441 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.579445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.579448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.579479 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.583293 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.585234 (+ 1941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.585238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.585240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.585350 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.585754 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.585758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.585760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.586017 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.586108 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.590262 (+ 4154us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.590474 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260620 12:14:13.590785 31256 ts_manager.cc:194] Re-registered known tserver with Master: 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599)
I20260620 12:14:13.591142 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.567870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.24.141.1:35577)
0620 12:14:13.568355 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.568369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.568503 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.569117 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.569138 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.569858 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.570942 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.570951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.572216 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.572220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.573694 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.573707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.574520 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.574525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.574914 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.575802 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.575830 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.579251 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.583416 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.583419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.583421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.583686 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.585506 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.585510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.585512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.585616 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.588639 (+ 3023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.588642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.588831 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.588939 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.590969 (+ 2030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":710,"spinlock_wait_cycles":384,"thread_start_us":98,"threads_started":1}
I20260620 12:14:13.591600 31256 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:50925
I20260620 12:14:13.591998 32221 heartbeater.cc:347] Connected to a master server at 127.24.141.60:41893
I20260620 12:14:13.592109 32221 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:13.592387 32221 heartbeater.cc:511] Master 127.24.141.60:41893 requested a full tablet report, sending...
I20260620 12:14:13.593990 32033 ts_manager.cc:194] Registered new tserver with Master: 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599)
I20260620 12:14:13.606410 31716 heartbeater.cc:503] Master 127.24.141.62:34705 was elected leader, sending a full tablet report...
W20260620 12:14:13.723517 32229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:13.723852 32229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:13.723901 32229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:13.728189 32229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:13.728271 32229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:13.728343 32229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:13.734566 32229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab
--principal=oryx/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:44783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=40253
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:13.736047 32229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:13.737373 32229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:13.749411 32237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:13.749668 32229 server_base.cc:1034] running on GCE node
W20260620 12:14:13.749459 32234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.749460 32235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:13.750353 32229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:13.751164 32229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:13.752379 32229 hybrid_clock.cc:648] HybridClock initialized: now 1781957653752350 us; error 52 us; skew 500 ppm
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:13.756222 32229 init.cc:377] Logged in from keytab as oryx/127.24.141.2@KRBTEST.COM (short username oryx)
I20260620 12:14:13.757462 32229 webserver.cc:460] Webserver started at http://127.24.141.2:40253/ using document root <none> and password file <none>
I20260620 12:14:13.758193 32229 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:13.758289 32229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:13.763427 32229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:14:13.766700 32244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.768688 32229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:13.768885 32229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "35e64331a4474036b783b1ec315708c2"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "34568c16267419b3ccc595ed629812d9"
server_key_iv: "a7999054169f2033f79d06b41e5cf4f7"
server_key_version: "encryptionkey@0"
I20260620 12:14:13.769631 32229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:13.795531 32229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:13.799392 32229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:13.799614 32229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:13.800318 32229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:13.801642 32229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:13.801703 32229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.801767 32229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:13.801795 32229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:13.812426 32229 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:44783
I20260620 12:14:13.812503 32357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:44783 every 8 connection(s)
I20260620 12:14:13.813766 32229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:13.814353 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 32229
I20260620 12:14:13.814574 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 31587
I20260620 12:14:13.824002 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:40285
--local_ip_for_outbound_sockets=127.24.141.3
--tserver_master_addrs=127.24.141.62:34705
--webserver_port=45129
--webserver_interface=127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab
--principal=oryx/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.60@KRBTEST.COM
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.61@KRBTEST.COM
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:13.836421 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.820378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:39537 (local address 127.24.141.60:41893)
0620 12:14:13.820562 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.820577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.820597 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:14:13.821759 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.821763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.821837 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.821891 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.823648 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.824307 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.826122 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.826462 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.826542 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.831593 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.831598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.831601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.831633 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.833262 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.834184 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.834189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.834191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.834285 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.834698 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.834701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.834703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.834954 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.835053 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.836043 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.836211 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:14:13.836421 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.818100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:44747 (local address 127.24.141.61:37555)
0620 12:14:13.818252 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.818257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.819896 (+ 1639us) server_negotiation.cc:407] Connection header received
0620 12:14:13.820934 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.820938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.820995 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.821050 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.823728 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.824441 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.826186 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.826518 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.826618 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.831661 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.831667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.831670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.831709 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.833537 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.834312 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.834316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.834318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.834430 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.834719 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.834722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.834724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.834964 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.835053 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.836043 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.836216 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:14:13.836584 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.817014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:50351 (local address 127.24.141.62:34705)
0620 12:14:13.817228 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:13.817233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:13.818828 (+ 1595us) server_negotiation.cc:407] Connection header received
0620 12:14:13.820840 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:13.820845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:13.820917 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:13.820979 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:13.825338 (+ 4359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.826099 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.827232 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.827966 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.828047 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.831336 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:13.831341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:13.831343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:13.831369 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:13.833808 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.834650 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.834657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.834659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.834822 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:13.835283 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:13.835288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:13.835290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:13.835620 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:13.835728 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:13.836188 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 12:14:13.836364 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 12:14:13.838800 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.819874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.24.141.2:39537)
0620 12:14:13.820202 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.820219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.821573 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.822065 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.822078 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.822419 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.823465 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.823502 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.824480 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.824483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.825984 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.825994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.826793 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.826796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.827036 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.827915 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.827946 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.831409 (+ 3463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.833432 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.833440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.833463 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.833857 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.834393 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.834396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.834398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.834589 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.835132 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.835135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.835602 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.837236 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.837476 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":417,"spinlock_wait_cycles":18944,"thread_start_us":123,"threads_started":1}
I20260620 12:14:13.839093 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.817841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:50351)
0620 12:14:13.818621 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.818682 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.820473 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.821578 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.821590 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.822388 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.825151 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.825169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.826231 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.826234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.827095 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.827105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.828219 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.828222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.828331 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.828762 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.828776 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.831116 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.833947 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.833950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.833952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.834469 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.834991 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.835034 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.835036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.835153 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.835825 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.835828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.835955 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.836778 (+ 823us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.837390 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":313,"spinlock_wait_cycles":25856,"thread_start_us":220,"threads_started":1}
I20260620 12:14:13.839313 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:13.817841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.24.141.2:44747)
0620 12:14:13.819780 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:13.819799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:13.820626 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:13.821581 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:13.821590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:13.822388 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:13.823466 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.823498 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.824646 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.824650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.826000 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:13.826015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:13.826843 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:13.826847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:13.827052 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:13.827913 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:13.827952 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:13.831132 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:13.833760 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.833764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.833767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.834173 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.834539 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:13.834542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:13.834544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:13.834624 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:13.835126 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:13.835134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:13.835602 (+ 468us) client_negotiation.cc:769] Sending connection context
0620 12:14:13.836725 (+ 1123us) client_negotiation.cc:240] Negotiation successful
0620 12:14:13.837241 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"mutex_wait_us":1547,"spinlock_wait_cycles":24192,"thread_start_us":189,"threads_started":1}
I20260620 12:14:13.840601 32359 heartbeater.cc:347] Connected to a master server at 127.24.141.62:34705
I20260620 12:14:13.840893 32359 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:13.841593 32359 heartbeater.cc:511] Master 127.24.141.62:34705 requested a full tablet report, sending...
I20260620 12:14:13.842159 32358 heartbeater.cc:347] Connected to a master server at 127.24.141.60:41893
I20260620 12:14:13.842247 32358 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:13.842434 32358 heartbeater.cc:511] Master 127.24.141.60:41893 requested a full tablet report, sending...
I20260620 12:14:13.842628 32360 heartbeater.cc:347] Connected to a master server at 127.24.141.61:37555
I20260620 12:14:13.842692 32360 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:13.842841 32360 heartbeater.cc:511] Master 127.24.141.61:37555 requested a full tablet report, sending...
I20260620 12:14:13.842931 31256 ts_manager.cc:194] Re-registered known tserver with Master: 35e64331a4474036b783b1ec315708c2 (127.24.141.2:44783)
I20260620 12:14:13.843721 31860 ts_manager.cc:194] Registered new tserver with Master: 35e64331a4474036b783b1ec315708c2 (127.24.141.2:44783)
I20260620 12:14:13.843719 31256 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:50351
I20260620 12:14:13.844327 32033 ts_manager.cc:194] Registered new tserver with Master: 35e64331a4474036b783b1ec315708c2 (127.24.141.2:44783)
W20260620 12:14:13.961747 32365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:13.962057 32365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:13.962105 32365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:13.966379 32365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:13.966470 32365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:13.966533 32365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:13.972497 32365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab
--principal=oryx/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:40285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=45129
--tserver_master_addrs=127.24.141.62:34705,127.24.141.61:37555,127.24.141.60:41893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:13.973799 32365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:13.974947 32365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:13.986080 32371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.986106 32373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:13.986080 32370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:13.986420 32365 server_base.cc:1034] running on GCE node
I20260620 12:14:13.987061 32365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:13.987811 32365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:13.989008 32365 hybrid_clock.cc:648] HybridClock initialized: now 1781957653988962 us; error 56 us; skew 500 ppm
Jun 20 12:14:13 dist-test-slave-rh5v krb5kdc[31212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:13.992662 32365 init.cc:377] Logged in from keytab as oryx/127.24.141.3@KRBTEST.COM (short username oryx)
I20260620 12:14:13.993944 32365 webserver.cc:460] Webserver started at http://127.24.141.3:45129/ using document root <none> and password file <none>
I20260620 12:14:13.994616 32365 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:13.994716 32365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:13.999981 32365 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260620 12:14:14.003019 32381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:14.005043 32365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 12:14:14.005293 32365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "4f8bbe4ed35c4b8a944802876b843431"
format_stamp: "Formatted at 2026-06-20 12:14:12 on dist-test-slave-rh5v"
server_key: "49ed7a0fc74181582143b7e929c68806"
server_key_iv: "5a4ca48af016ce92d080e89a91c7a67b"
server_key_version: "encryptionkey@0"
I20260620 12:14:14.006047 32365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:14.006443 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.001613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.0.0.1:52616)
0620 12:14:14.001891 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.001906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.002047 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.002462 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.002464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.002485 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:14.002622 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.002630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.003757 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.003761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.004965 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.004982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.005895 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.005899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.006010 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.006050 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.006157 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.006269 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260620 12:14:14.006474 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.001705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.24.141.60:41893)
0620 12:14:14.001840 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.001844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.002017 (+ 173us) server_negotiation.cc:407] Connection header received
0620 12:14:14.002232 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.002235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.002306 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.002425 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:14.002769 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.003611 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.005182 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.005735 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.005837 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.005992 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.006200 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.006307 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:14:14.007292 32051 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 12:14:14.007918 32382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49e4418976f48feace83ae97c424b7c. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:14.008106 32382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: This master's current role is: LEARNER
I20260620 12:14:14.022061 32365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:14.025758 32365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:14.025971 32365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:14.026626 32365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:14.027604 32365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:14.027655 32365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:14.027716 32365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:14.027745 32365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:14.038399 32365 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:40285
I20260620 12:14:14.038417 32495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:40285 every 8 connection(s)
I20260620 12:14:14.039765 32365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:14.043185 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 32365
Jun 20 12:14:14 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
Jun 20 12:14:14 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.61@KRBTEST.COM
Jun 20 12:14:14 dist-test-slave-rh5v krb5kdc[31212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957653, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.60@KRBTEST.COM
I20260620 12:14:14.060809 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.045466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.24.141.62:34705)
0620 12:14:14.045710 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.045714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.047778 (+ 2064us) server_negotiation.cc:407] Connection header received
0620 12:14:14.047986 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.047989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.048049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.048116 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.049229 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.049997 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.051245 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.052153 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.052258 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.054866 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.054872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.054875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.054909 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.056858 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.057526 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.057531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.057533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.057643 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.057992 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.057996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.057998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.059384 (+ 1386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.059533 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.060457 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.060603 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":521,"server-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
I20260620 12:14:14.062228 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.045156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:40417 (local address 127.24.141.62:34705)
0620 12:14:14.045309 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.045314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.045332 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:14:14.046230 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.046235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.046293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.046375 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.048571 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.049487 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.051589 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.051952 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.052031 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.056154 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.056159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.056161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.056194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.057779 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.059464 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.059469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.059471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.059573 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.060123 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.060127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.060129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.060365 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.060447 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.061865 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.062030 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":72}
I20260620 12:14:14.062878 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.042240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:35937 (local address 127.24.141.60:41893)
0620 12:14:14.042588 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.042592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.044567 (+ 1975us) server_negotiation.cc:407] Connection header received
0620 12:14:14.046252 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.046256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.046310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.046364 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.048745 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.049566 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.051588 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.051950 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.052031 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.057500 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.057504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.057507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.057540 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.059099 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.061422 (+ 2323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.061426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.061428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.061534 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.061866 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.061870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.061871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.062092 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.062178 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.062546 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.062699 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260620 12:14:14.064898 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.043687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:40417)
0620 12:14:14.044626 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.044644 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.045957 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.046733 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.046746 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.047475 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:14.048381 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.048407 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.049637 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.049640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.051415 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.051429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.052395 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.052398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.052574 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.053686 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:14.053696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:14.055904 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:14.058136 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:14.058145 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:14.058168 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:14.058769 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:14.059710 (+ 941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:14.059713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:14.059715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:14.060000 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:14.060696 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:14.060706 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:14.061515 (+ 809us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.062727 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.063412 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":495,"spinlock_wait_cycles":61184,"thread_start_us":141,"threads_started":1}
I20260620 12:14:14.065002 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.042792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.24.141.3:35937)
0620 12:14:14.044320 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.044416 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.045953 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.046736 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.046746 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.047639 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:14.048595 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.048605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.049683 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.049686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.051415 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.051430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.052261 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.052265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.052577 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.053570 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:14.053583 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:14.057298 (+ 3715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:14.061002 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:14.061006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:14.061008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:14.061293 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:14.061654 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:14.061657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:14.061659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:14.061764 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:14.062238 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:14.062241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:14.062344 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.063042 (+ 698us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.063771 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":818,"spinlock_wait_cycles":18688,"thread_start_us":690,"threads_started":1}
I20260620 12:14:14.066911 32496 heartbeater.cc:347] Connected to a master server at 127.24.141.60:41893
I20260620 12:14:14.067142 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.046592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.24.141.60:41893)
0620 12:14:14.047149 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.047153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.053929 (+ 6776us) server_negotiation.cc:407] Connection header received
0620 12:14:14.053998 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.054001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.054069 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.054166 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.055623 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.056635 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.058735 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.059101 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.059187 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.062245 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.062250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.062253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.062279 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.064140 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.064619 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.064623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.064625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.064708 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.064982 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.064986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.064988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.065257 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.065381 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.066041 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.066970 (+ 929us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":103,"threads_started":1}
I20260620 12:14:14.067238 32496 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:14.068011 32496 heartbeater.cc:511] Master 127.24.141.60:41893 requested a full tablet report, sending...
I20260620 12:14:14.068650 32497 heartbeater.cc:347] Connected to a master server at 127.24.141.62:34705
I20260620 12:14:14.068749 32497 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:14.068964 32497 heartbeater.cc:511] Master 127.24.141.62:34705 requested a full tablet report, sending...
I20260620 12:14:14.065167 31256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:14:14.069636 32033 ts_manager.cc:194] Registered new tserver with Master: 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3:40285)
I20260620 12:14:14.069968 31257 ts_manager.cc:194] Re-registered known tserver with Master: 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3:40285)
W20260620 12:14:14.070648 31256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:14.070703 31257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:40417
I20260620 12:14:14.074199 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: NON_VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: true } } }
I20260620 12:14:14.074371 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:14.075876 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.044773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:56855 (local address 127.24.141.61:37555)
0620 12:14:14.044900 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.044916 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.046228 (+ 1312us) server_negotiation.cc:407] Connection header received
0620 12:14:14.046419 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.046423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.046482 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.046538 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.048554 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.049395 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.051502 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.051803 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.051882 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.060619 (+ 8737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.060623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.060626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.060656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.067499 (+ 6843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.073410 (+ 5911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.073416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.073418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.073577 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.074884 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.074887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.074889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.075132 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.075213 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.075558 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.075703 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:14:14.076503 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.045000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.24.141.3:56855)
0620 12:14:14.046142 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.046156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.046295 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.046735 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.046747 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.047795 (+ 1048us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:14.048380 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.048405 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.049573 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.049577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.051373 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.051383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.052260 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.052264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.052566 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.053377 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:14.053402 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:14.060426 (+ 7024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:14.072849 (+ 12423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:14.072853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:14.072856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:14.073222 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:14.074561 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:14.074565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:14.074567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:14.074767 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:14.075282 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:14.075286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:14.075383 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.076037 (+ 654us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.076296 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":207,"spinlock_wait_cycles":1408,"thread_start_us":254,"threads_started":1}
I20260620 12:14:14.077127 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.045712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.24.141.61:37555)
0620 12:14:14.046017 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.046021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.052175 (+ 6154us) server_negotiation.cc:407] Connection header received
0620 12:14:14.052213 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.052216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.052263 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.052351 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.056858 (+ 4507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.057647 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.058735 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.059095 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.059187 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.061005 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.061009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.061012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.061037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.073870 (+ 12833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.074765 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.074771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.074773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.074884 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.075181 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.075184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.075185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.075358 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.075478 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.076515 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.076929 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
I20260620 12:14:14.077764 32498 heartbeater.cc:347] Connected to a master server at 127.24.141.61:37555
I20260620 12:14:14.077878 32498 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:14.078116 32498 heartbeater.cc:511] Master 127.24.141.61:37555 requested a full tablet report, sending...
I20260620 12:14:14.078853 32515 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: attempting to promote NON_VOTER e4ec80927b884b3d8a46fd7d9601f85e to VOTER
I20260620 12:14:14.079092 31860 ts_manager.cc:194] Registered new tserver with Master: 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3:40285)
I20260620 12:14:14.079227 32515 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:14.083730 32051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 LEARNER]: Refusing update from remote peer b49e4418976f48feace83ae97c424b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260620 12:14:14.083886 31870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Refusing update from remote peer b49e4418976f48feace83ae97c424b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260620 12:14:14.084336 32514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 12:14:14.084988 32515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 12:14:14.087301 32515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.087893 31870 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.088434 32051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.090626 32523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.090773 32523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:14:14.090867 32382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.090991 32382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:14:14.093989 32514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.094143 32514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:14.094276 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.094374 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:14.094522 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b49e4418976f48feace83ae97c424b7c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:14.094610 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:14.098534 32527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.092339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:40285 (local address 127.0.0.1:49558)
0620 12:14:14.093435 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.093449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.093559 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.094162 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.094165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.094183 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:14.094590 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.094599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.095776 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.095779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.097031 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.097042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.098075 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.098080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.098167 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.098202 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.098274 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.098365 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":102,"threads_started":1}
I20260620 12:14:14.098668 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.092514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.24.141.3:40285)
0620 12:14:14.092835 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.092847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.093553 (+ 706us) server_negotiation.cc:407] Connection header received
0620 12:14:14.093685 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.093703 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.093958 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.094118 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:14.094751 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.095655 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.097170 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.097941 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.098047 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.098179 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.098359 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.098514 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
I20260620 12:14:14.099021 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.092031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:39599 (local address 127.0.0.1:45122)
0620 12:14:14.092165 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.092181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.092271 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.093497 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.093501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.093526 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:14.093663 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.093670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.095270 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.095274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.096240 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.096250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.098523 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.098528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.098630 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.098666 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.098734 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.098843 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260620 12:14:14.099887 32525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.093043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:44783 (local address 127.0.0.1:40354)
0620 12:14:14.094837 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.094853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.094966 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.095531 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.095534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.095553 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:14.095686 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.095692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.097111 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.097115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.098115 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.098123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.099497 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.099501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.099594 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.099619 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.099686 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.099765 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1720,"thread_start_us":118,"threads_started":1}
I20260620 12:14:14.100153 32528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.093243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.24.141.2:44783)
0620 12:14:14.094527 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.094540 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.095068 (+ 528us) server_negotiation.cc:407] Connection header received
0620 12:14:14.095169 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.095174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.095389 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.095544 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:14.095803 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.096957 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.098275 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.099362 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.099487 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.099647 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.099725 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.099840 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":129,"threads_started":1}
I20260620 12:14:14.100409 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.092270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.24.141.1:39599)
0620 12:14:14.092773 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.092782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.092813 (+ 31us) server_negotiation.cc:407] Connection header received
0620 12:14:14.092922 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.092929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.093263 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.093834 (+ 571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:14.093984 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.095146 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.096395 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.098374 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.099752 (+ 1378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.099904 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.100151 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.100274 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":135,"threads_started":1}
I20260620 12:14:14.102744 32430 tablet_service.cc:1467] Processing CreateTablet for tablet fea876cd38c841a8be7a20638c11f878 (DEFAULT_TABLE table=test-table [id=6b86b0fb09034ce4b612fdbf1a94ff13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:14.103082 32155 tablet_service.cc:1467] Processing CreateTablet for tablet fea876cd38c841a8be7a20638c11f878 (DEFAULT_TABLE table=test-table [id=6b86b0fb09034ce4b612fdbf1a94ff13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:14.103263 32292 tablet_service.cc:1467] Processing CreateTablet for tablet fea876cd38c841a8be7a20638c11f878 (DEFAULT_TABLE table=test-table [id=6b86b0fb09034ce4b612fdbf1a94ff13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:14.104241 32155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea876cd38c841a8be7a20638c11f878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:14.104314 32430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea876cd38c841a8be7a20638c11f878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:14.104490 32292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea876cd38c841a8be7a20638c11f878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:14.115197 32529 tablet_bootstrap.cc:492] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Bootstrap starting.
I20260620 12:14:14.116942 32531 tablet_bootstrap.cc:492] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Bootstrap starting.
I20260620 12:14:14.117908 32529 tablet_bootstrap.cc:654] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:14.118966 32529 log.cc:824] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:14.119764 32531 tablet_bootstrap.cc:654] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:14.120582 32530 tablet_bootstrap.cc:492] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Bootstrap starting.
I20260620 12:14:14.121049 32531 log.cc:824] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:14.122746 32530 tablet_bootstrap.cc:654] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:14.123545 32530 log.cc:824] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:14.123752 32529 tablet_bootstrap.cc:492] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: No bootstrap required, opened a new log
I20260620 12:14:14.124002 32529 ts_tablet_manager.cc:1397] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 12:14:14.125903 32530 tablet_bootstrap.cc:492] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: No bootstrap required, opened a new log
I20260620 12:14:14.126097 32530 ts_tablet_manager.cc:1397] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:14:14.127813 32531 tablet_bootstrap.cc:492] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: No bootstrap required, opened a new log
I20260620 12:14:14.128096 32531 ts_tablet_manager.cc:1397] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 12:14:14.128353 32529 raft_consensus.cc:348] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.128695 32529 raft_consensus.cc:374] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:14.128754 32529 raft_consensus.cc:729] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d53916f670f446b86f725d97b7fbb3c, State: Initialized, Role: FOLLOWER
I20260620 12:14:14.129348 32529 consensus_queue.cc:260] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.130767 32529 ts_tablet_manager.cc:1428] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 12:14:14.131080 32222 heartbeater.cc:503] Master 127.24.141.62:34705 was elected leader, sending a full tablet report...
I20260620 12:14:14.131160 32530 raft_consensus.cc:348] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.131609 32530 raft_consensus.cc:374] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:14.131669 32530 raft_consensus.cc:729] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8bbe4ed35c4b8a944802876b843431, State: Initialized, Role: FOLLOWER
I20260620 12:14:14.134037 32531 raft_consensus.cc:348] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.134413 32531 raft_consensus.cc:374] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:14.134469 32531 raft_consensus.cc:729] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e64331a4474036b783b1ec315708c2, State: Initialized, Role: FOLLOWER
I20260620 12:14:14.135061 32531 consensus_queue.cc:260] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.135357 32530 consensus_queue.cc:260] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.136291 32497 heartbeater.cc:503] Master 127.24.141.62:34705 was elected leader, sending a full tablet report...
I20260620 12:14:14.136349 32531 ts_tablet_manager.cc:1428] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 12:14:14.137859 32359 heartbeater.cc:503] Master 127.24.141.62:34705 was elected leader, sending a full tablet report...
I20260620 12:14:14.140522 32530 ts_tablet_manager.cc:1428] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260620 12:14:14.201370 32537 raft_consensus.cc:482] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:14.201661 32537 raft_consensus.cc:504] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.203421 32537 leader_election.cc:288] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599), 35e64331a4474036b783b1ec315708c2 (127.24.141.2:44783)
I20260620 12:14:14.208889 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.203810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:45537 (local address 127.24.141.1:39599)
0620 12:14:14.203979 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.203984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.204007 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:14.204202 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.204207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.204265 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.204355 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:14.204704 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.205545 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.207468 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.208140 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.208299 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.208356 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.208622 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.208738 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:14:14.208889 32528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.204081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:58381 (local address 127.24.141.2:44783)
0620 12:14:14.204271 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.204276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.204292 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:14:14.204345 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.204347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.204391 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.204480 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:14.204703 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.205545 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.207468 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.208132 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.208299 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.208357 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.208622 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.208729 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 12:14:14.208958 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.203686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:39599 (local address 127.24.141.3:45537)
0620 12:14:14.203876 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.203894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.204015 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.204390 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.204393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.204407 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:14.204575 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.204583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.205707 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.205712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.207272 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.207285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.208308 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.208313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.208500 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.208516 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.208584 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.208725 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":182}
I20260620 12:14:14.208958 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.203881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:44783 (local address 127.24.141.3:58381)
0620 12:14:14.204127 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:14.204143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:14.204242 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:14.204492 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:14.204494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:14.204507 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:14.204589 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.204594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.205707 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.205712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.207272 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:14.207285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.208308 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.208313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:14.208448 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.208496 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 12:14:14.208584 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:14:14.208725 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":199}
I20260620 12:14:14.209975 32175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea876cd38c841a8be7a20638c11f878" candidate_uuid: "4f8bbe4ed35c4b8a944802876b843431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d53916f670f446b86f725d97b7fbb3c" is_pre_election: true
I20260620 12:14:14.209980 32312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea876cd38c841a8be7a20638c11f878" candidate_uuid: "4f8bbe4ed35c4b8a944802876b843431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e64331a4474036b783b1ec315708c2" is_pre_election: true
I20260620 12:14:14.210420 32175 raft_consensus.cc:2454] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8bbe4ed35c4b8a944802876b843431 in term 0.
I20260620 12:14:14.210446 32312 raft_consensus.cc:2454] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8bbe4ed35c4b8a944802876b843431 in term 0.
I20260620 12:14:14.211023 32385 leader_election.cc:302] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d53916f670f446b86f725d97b7fbb3c, 35e64331a4474036b783b1ec315708c2, 4f8bbe4ed35c4b8a944802876b843431; no voters:
I20260620 12:14:14.211357 32537 raft_consensus.cc:2781] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:14.211517 32537 raft_consensus.cc:482] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:14.211584 32537 raft_consensus.cc:3037] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:14.213032 32537 raft_consensus.cc:504] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.213575 32537 leader_election.cc:288] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [CANDIDATE]: Term 1 election: Requested vote from peers 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599), 35e64331a4474036b783b1ec315708c2 (127.24.141.2:44783)
I20260620 12:14:14.214061 32312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea876cd38c841a8be7a20638c11f878" candidate_uuid: "4f8bbe4ed35c4b8a944802876b843431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e64331a4474036b783b1ec315708c2"
I20260620 12:14:14.214118 32175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea876cd38c841a8be7a20638c11f878" candidate_uuid: "4f8bbe4ed35c4b8a944802876b843431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d53916f670f446b86f725d97b7fbb3c"
I20260620 12:14:14.214260 32312 raft_consensus.cc:3037] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:14.214260 32175 raft_consensus.cc:3037] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:14.215947 32175 raft_consensus.cc:2454] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8bbe4ed35c4b8a944802876b843431 in term 1.
I20260620 12:14:14.215947 32312 raft_consensus.cc:2454] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8bbe4ed35c4b8a944802876b843431 in term 1.
I20260620 12:14:14.216451 32383 leader_election.cc:302] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d53916f670f446b86f725d97b7fbb3c, 4f8bbe4ed35c4b8a944802876b843431; no voters:
I20260620 12:14:14.216701 32537 raft_consensus.cc:2781] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:14.216977 32537 raft_consensus.cc:686] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8bbe4ed35c4b8a944802876b843431, State: Running, Role: LEADER
I20260620 12:14:14.217430 32537 consensus_queue.cc:237] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } }
I20260620 12:14:14.221915 31256 catalog_manager.cc:5495] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "4f8bbe4ed35c4b8a944802876b843431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } health_report { overall_health: UNKNOWN } } }
I20260620 12:14:14.261768 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.257824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.24.141.3:40285)
0620 12:14:14.257997 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.258003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.258021 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:14:14.258167 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.258170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.258245 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.258369 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:14.258621 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.259190 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.260113 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.260330 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.260424 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.260740 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:14.261342 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:14.261464 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.261530 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.261596 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:14:14.270030 32175 raft_consensus.cc:1261] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 1 FOLLOWER]: Refusing update from remote peer 4f8bbe4ed35c4b8a944802876b843431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:14.270103 32312 raft_consensus.cc:1261] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 1 FOLLOWER]: Refusing update from remote peer 4f8bbe4ed35c4b8a944802876b843431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:14.270963 32538 consensus_queue.cc:1035] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:14.271247 32537 consensus_queue.cc:1035] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:14.296516 31256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:14.296773 31256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:14.302738 31256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b49e4418976f48feace83ae97c424b7c: Sending DeleteTablet for 3 replicas of tablet fea876cd38c841a8be7a20638c11f878
I20260620 12:14:14.303933 32155 tablet_service.cc:1514] Processing DeleteTablet for tablet fea876cd38c841a8be7a20638c11f878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:14 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45122
I20260620 12:14:14.304352 32292 tablet_service.cc:1514] Processing DeleteTablet for tablet fea876cd38c841a8be7a20638c11f878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:14 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:40354
I20260620 12:14:14.304653 32550 tablet_replica.cc:330] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: stopping tablet replica
I20260620 12:14:14.304956 32430 tablet_service.cc:1514] Processing DeleteTablet for tablet fea876cd38c841a8be7a20638c11f878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:14 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:49558
I20260620 12:14:14.304955 32551 tablet_replica.cc:330] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: stopping tablet replica
I20260620 12:14:14.305231 32550 raft_consensus.cc:2229] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:14.305656 32551 raft_consensus.cc:2229] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:14.305711 32550 raft_consensus.cc:2258] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:14.306344 32551 raft_consensus.cc:2258] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:14.308143 32550 ts_tablet_manager.cc:1905] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:14.309789 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 31225
I20260620 12:14:14.313575 32550 ts_tablet_manager.cc:1918] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:14.313721 32550 log.cc:1196] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/fea876cd38c841a8be7a20638c11f878
I20260620 12:14:14.314117 32550 ts_tablet_manager.cc:1939] T fea876cd38c841a8be7a20638c11f878 P 1d53916f670f446b86f725d97b7fbb3c: Deleting consensus metadata
I20260620 12:14:14.318643 32551 ts_tablet_manager.cc:1905] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 12:14:14.319046 32109 connection.cc:535] server connection from 127.0.0.1:45122 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:14:14.321050 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.308871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.24.141.61:37555)
0620 12:14:14.309004 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.309008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.309024 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:14:14.309112 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.309115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.309173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.309260 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.311723 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.312565 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.314072 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.314440 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.314522 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.316994 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.316999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.317002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.317037 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.318700 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.319483 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.319487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.319489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.319588 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.319973 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.319977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.319978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.320217 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.320318 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.320698 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.320877 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:14:14.322618 32551 ts_tablet_manager.cc:1918] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:14.322739 32551 log.cc:1196] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/fea876cd38c841a8be7a20638c11f878
I20260620 12:14:14.323123 32551 ts_tablet_manager.cc:1939] T fea876cd38c841a8be7a20638c11f878 P 35e64331a4474036b783b1ec315708c2: Deleting consensus metadata
I20260620 12:14:14.324821 32552 tablet_replica.cc:330] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: stopping tablet replica
I20260620 12:14:14.325297 32552 raft_consensus.cc:2229] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 1 LEADER]: Raft consensus shutting down.
W20260620 12:14:14.325410 32247 connection.cc:441] server connection from 127.0.0.1:40354 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40354 (request call id 1) could send its response
I20260620 12:14:14.325783 32552 raft_consensus.cc:2258] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:14.327719 32552 ts_tablet_manager.cc:1905] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:14.331167 32552 ts_tablet_manager.cc:1918] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:14.331285 32552 log.cc:1196] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/fea876cd38c841a8be7a20638c11f878
I20260620 12:14:14.331643 32552 ts_tablet_manager.cc:1939] T fea876cd38c841a8be7a20638c11f878 P 4f8bbe4ed35c4b8a944802876b843431: Deleting consensus metadata
W20260620 12:14:14.333000 32383 connection.cc:441] server connection from 127.0.0.1:49558 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49558 (request call id 1) could send its response
I20260620 12:14:14.333841 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.309014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.24.141.60:41893)
0620 12:14:14.309149 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.309153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.323194 (+ 14041us) server_negotiation.cc:407] Connection header received
0620 12:14:14.323377 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.323380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.323446 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.323526 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.324461 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.325196 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.326593 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.326935 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.327054 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.328838 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.328843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.328846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.328879 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.330897 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.331742 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.331746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.331747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.331856 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.332406 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.332410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.332411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.332667 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.332802 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.333513 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.333660 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:14:14.337189 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.325600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.24.141.61:37555)
0620 12:14:14.325723 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.325727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.326225 (+ 498us) server_negotiation.cc:407] Connection header received
0620 12:14:14.326391 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.326394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.326440 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.326490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.328580 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.329363 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.330765 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.331195 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.331303 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.333825 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.333830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.333833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.333863 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.335533 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.335952 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.335955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.335957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.336037 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.336290 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.336293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.336295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.336486 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.336597 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.336861 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.336975 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:14:14.344376 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:14.332486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.24.141.60:41893)
0620 12:14:14.332618 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:14.332622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:14.334106 (+ 1484us) server_negotiation.cc:407] Connection header received
0620 12:14:14.334307 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:14.334310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:14.334367 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:14.334421 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:14.335370 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.336243 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.338134 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:14.338485 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:14.338584 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:14.340384 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:14.340388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:14.340390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:14.340417 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:14.342118 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.342706 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.342710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.342712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.342816 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:14.343169 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:14.343172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:14.343174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:14.343428 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:14.343582 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:14.343839 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 12:14:14.343990 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:14:15.289812 32560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.289133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:57423)
0620 12:14:15.289498 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.289617 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":98,"threads_started":1}
I20260620 12:14:15.290107 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.289435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:50131)
0620 12:14:15.289869 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.289955 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":109,"threads_started":1}
W20260620 12:14:15.290220 32359 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:34705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
W20260620 12:14:15.290343 32497 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:34705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:15.292273 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.291715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:60319)
0620 12:14:15.292054 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.292156 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":88,"threads_started":1}
W20260620 12:14:15.292640 32222 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:34705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:15.803334 32563 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b49e4418976f48feace83ae97c424b7c)
I20260620 12:14:15.803565 32564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b49e4418976f48feace83ae97c424b7c)
I20260620 12:14:15.803637 32563 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:15.803799 32564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:15.805017 32563 leader_election.cc:288] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705), 8c59ee4a783841f7b5ed08b90271dc62 (127.24.141.61:37555)
I20260620 12:14:15.804986 32564 leader_election.cc:288] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705), e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60:41893)
I20260620 12:14:15.805893 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.805213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:52922)
0620 12:14:15.805625 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.805730 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":94,"threads_started":1}
I20260620 12:14:15.805891 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.805279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:52930)
0620 12:14:15.805666 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.805760 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":100,"threads_started":1}
W20260620 12:14:15.806063 31847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
W20260620 12:14:15.806115 32029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:15.807232 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.806939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:52940)
0620 12:14:15.807088 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.807149 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 12:14:15.807349 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.806946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:52954)
0620 12:14:15.807184 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.807235 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
W20260620 12:14:15.807474 31847 leader_election.cc:334] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705): Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
W20260620 12:14:15.807514 32029 leader_election.cc:334] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705): Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:15.811213 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.805327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:37555 (local address 127.0.0.1:45204)
0620 12:14:15.806179 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.806196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:15.806348 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:15.806737 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:15.806740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:15.806759 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:15.806871 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:15.806878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.807958 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.807961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:15.809499 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:15.809514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.810606 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.810610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:15.810747 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:15.810774 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:14:15.810867 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 12:14:15.811033 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":33,"thread_start_us":69,"threads_started":1}
I20260620 12:14:15.811295 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.805636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.24.141.61:37555)
0620 12:14:15.806022 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:15.806026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:15.806440 (+ 414us) server_negotiation.cc:407] Connection header received
0620 12:14:15.806491 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:15.806494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:15.806540 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:15.806618 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:15.806987 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.807827 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.809709 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.810422 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.810571 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:15.810670 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:15.810944 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 12:14:15.811092 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":319,"thread_start_us":65,"threads_started":1}
I20260620 12:14:15.811293 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.805906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.24.141.60:41893)
0620 12:14:15.806216 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:15.806221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:15.806622 (+ 401us) server_negotiation.cc:407] Connection header received
0620 12:14:15.806787 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:15.806790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:15.806828 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:15.806883 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:15.807216 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.807975 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.809709 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.810422 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.810571 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:15.810675 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:15.811022 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 12:14:15.811121 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":207,"thread_start_us":62,"threads_started":1}
I20260620 12:14:15.811651 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.805494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:41893 (local address 127.0.0.1:52660)
0620 12:14:15.806543 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.806556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:15.806686 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:15.806993 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:15.806996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:15.807012 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:15.807097 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:15.807102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.808088 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.808092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:15.809499 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:15.809513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:15.810606 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:15.810610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:15.810742 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:15.810775 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 12:14:15.810983 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 12:14:15.811148 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":52,"thread_start_us":73,"threads_started":1}
I20260620 12:14:15.811823 31870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c59ee4a783841f7b5ed08b90271dc62" is_pre_election: true
I20260620 12:14:15.812136 31870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ec80927b884b3d8a46fd7d9601f85e in term 1.
I20260620 12:14:15.812273 32051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c59ee4a783841f7b5ed08b90271dc62" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" is_pre_election: true
I20260620 12:14:15.812544 32051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c59ee4a783841f7b5ed08b90271dc62 in term 1.
I20260620 12:14:15.812633 32026 leader_election.cc:302] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c59ee4a783841f7b5ed08b90271dc62, e4ec80927b884b3d8a46fd7d9601f85e; no voters: b49e4418976f48feace83ae97c424b7c
I20260620 12:14:15.812881 32563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 12:14:15.812942 32563 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Starting leader election (detected failure of leader b49e4418976f48feace83ae97c424b7c)
I20260620 12:14:15.812986 32563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:14:15.813340 31844 leader_election.cc:302] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c59ee4a783841f7b5ed08b90271dc62, e4ec80927b884b3d8a46fd7d9601f85e; no voters: b49e4418976f48feace83ae97c424b7c
I20260620 12:14:15.813889 32564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 12:14:15.813951 32564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b49e4418976f48feace83ae97c424b7c)
I20260620 12:14:15.813985 32564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:14:15.814462 32563 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:15.814926 32563 leader_election.cc:288] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [CANDIDATE]: Term 2 election: Requested vote from peers b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705), 8c59ee4a783841f7b5ed08b90271dc62 (127.24.141.61:37555)
I20260620 12:14:15.815423 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.815105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:52970)
0620 12:14:15.815236 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.815336 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 12:14:15.815367 32564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:15.815604 31870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c59ee4a783841f7b5ed08b90271dc62"
I20260620 12:14:15.815809 31870 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e4ec80927b884b3d8a46fd7d9601f85e in current term 2: Already voted for candidate 8c59ee4a783841f7b5ed08b90271dc62 in this term.
I20260620 12:14:15.816123 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.815855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:52978)
0620 12:14:15.815998 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.816056 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 12:14:15.816213 32029 leader_election.cc:334] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705): Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:15.816298 32029 leader_election.cc:302] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4ec80927b884b3d8a46fd7d9601f85e; no voters: 8c59ee4a783841f7b5ed08b90271dc62, b49e4418976f48feace83ae97c424b7c
I20260620 12:14:15.816329 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.816012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:52984)
0620 12:14:15.816201 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.816254 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 12:14:15.816419 32564 leader_election.cc:288] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 2 election: Requested vote from peers b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705), e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60:41893)
I20260620 12:14:15.816501 32563 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 12:14:15.817044 32051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c59ee4a783841f7b5ed08b90271dc62" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
I20260620 12:14:15.817219 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:15.816912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:53000)
0620 12:14:15.817076 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:15.817147 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:14:15.817301 32051 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c59ee4a783841f7b5ed08b90271dc62 in current term 2: Already voted for candidate e4ec80927b884b3d8a46fd7d9601f85e in this term.
W20260620 12:14:15.817344 31847 leader_election.cc:334] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705): Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:15.817778 31844 leader_election.cc:302] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c59ee4a783841f7b5ed08b90271dc62; no voters: b49e4418976f48feace83ae97c424b7c, e4ec80927b884b3d8a46fd7d9601f85e
I20260620 12:14:15.818003 32564 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 12:14:16.291527 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:16.290880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:39741)
0620 12:14:16.291302 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:16.291381 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":91,"threads_started":1}
I20260620 12:14:16.291527 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:16.290880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:60363)
0620 12:14:16.291303 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:16.291381 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":126,"threads_started":1}
I20260620 12:14:16.293919 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:16.293389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:45689)
0620 12:14:16.293730 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:16.293815 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":87,"threads_started":1}
I20260620 12:14:17.293229 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.292565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:43619)
0620 12:14:17.292977 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.293054 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":108,"threads_started":1}
I20260620 12:14:17.293229 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.292565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:59323)
0620 12:14:17.292977 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.293083 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":108,"threads_started":1}
I20260620 12:14:17.295202 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.294703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:48415)
0620 12:14:17.295033 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.295098 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":81,"threads_started":1}
I20260620 12:14:17.517987 32579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:17.518139 32579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:17.518628 32579 leader_election.cc:288] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705), e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60:41893)
I20260620 12:14:17.519299 32051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c59ee4a783841f7b5ed08b90271dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" is_pre_election: true
I20260620 12:14:17.519402 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.518826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:53218)
0620 12:14:17.519134 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.519229 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260620 12:14:17.519454 32051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c59ee4a783841f7b5ed08b90271dc62 in term 2.
I20260620 12:14:17.519912 31844 leader_election.cc:302] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c59ee4a783841f7b5ed08b90271dc62, e4ec80927b884b3d8a46fd7d9601f85e; no voters:
I20260620 12:14:17.520195 32579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 12:14:17.520275 32579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:17.520316 32579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:14:17.520349 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.520138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:53230)
0620 12:14:17.520230 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.520280 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 12:14:17.520493 31847 leader_election.cc:334] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705): Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:17.521600 32579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:17.522019 32579 leader_election.cc:288] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 3 election: Requested vote from peers b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705), e4ec80927b884b3d8a46fd7d9601f85e (127.24.141.60:41893)
I20260620 12:14:17.522491 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.522234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:53236)
0620 12:14:17.522337 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.522393 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 12:14:17.522506 32051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c59ee4a783841f7b5ed08b90271dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4ec80927b884b3d8a46fd7d9601f85e"
I20260620 12:14:17.522646 32051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:14:17.523159 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.522977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:53242)
0620 12:14:17.523067 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.523114 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 12:14:17.523247 31847 leader_election.cc:334] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705): Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
I20260620 12:14:17.523698 32051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c59ee4a783841f7b5ed08b90271dc62 in term 3.
I20260620 12:14:17.524094 31844 leader_election.cc:302] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c59ee4a783841f7b5ed08b90271dc62, e4ec80927b884b3d8a46fd7d9601f85e; no voters: b49e4418976f48feace83ae97c424b7c
I20260620 12:14:17.524303 32579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:14:17.524520 32579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [term 3 LEADER]: Becoming Leader. State: Replica: 8c59ee4a783841f7b5ed08b90271dc62, State: Running, Role: LEADER
I20260620 12:14:17.524788 32579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } }
I20260620 12:14:17.525897 32582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c59ee4a783841f7b5ed08b90271dc62. Latest consensus state: current_term: 3 leader_uuid: "8c59ee4a783841f7b5ed08b90271dc62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:17.526005 32582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:17.526324 32584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:17.528985 32584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:17.529454 32584 catalog_manager.cc:1259] Loaded cluster ID: 94a7ea63675846419b653abf3c562898
I20260620 12:14:17.529505 32584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:17.530189 32584 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:17.530586 32584 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Loaded TSK: 0
I20260620 12:14:17.531416 32584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:14:17.592523 31860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:14:17.596065 31860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:17.600288 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.599951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:53264)
0620 12:14:17.600117 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.600183 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 12:14:17.600368 32051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [term 3 FOLLOWER]: Refusing update from remote peer 8c59ee4a783841f7b5ed08b90271dc62: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 12:14:17.600950 32582 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 12:14:17.601171 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.600899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.0.0.1:53266)
0620 12:14:17.601003 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.601098 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 12:14:17.601300 31847 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 -> Peer b49e4418976f48feace83ae97c424b7c (127.24.141.62:34705): Couldn't send request to peer b49e4418976f48feace83ae97c424b7c. Status: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 12:14:17.602687 32586 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c59ee4a783841f7b5ed08b90271dc62. Latest consensus state: current_term: 3 leader_uuid: "8c59ee4a783841f7b5ed08b90271dc62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:17.602842 32586 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:14:17.604127 32585 mvcc.cc:205] Tried to move back new op lower bound from 7298898565523759104 to 7298898565224169472. Current Snapshot: MvccSnapshot[applied={T|T < 7298898565523759104}]
I20260620 12:14:17.604822 32586 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8c59ee4a783841f7b5ed08b90271dc62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:17.604980 32586 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ec80927b884b3d8a46fd7d9601f85e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:14:17.605639 32582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8c59ee4a783841f7b5ed08b90271dc62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b49e4418976f48feace83ae97c424b7c" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 34705 } } peers { permanent_uuid: "8c59ee4a783841f7b5ed08b90271dc62" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 37555 } attrs { promote: false } } peers { permanent_uuid: "e4ec80927b884b3d8a46fd7d9601f85e" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 41893 } attrs { promote: false } } }
I20260620 12:14:17.605788 32582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:17.618351 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.613103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.24.141.2:44783)
0620 12:14:17.614560 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:17.614565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:17.614579 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:14:17.614630 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:17.614633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:17.614681 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:17.615212 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:17.615323 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.615964 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.617149 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.617695 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.617817 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.617900 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:17.618073 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 12:14:17.618167 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1373,"thread_start_us":112,"threads_started":1}
I20260620 12:14:17.618345 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.612912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:44783 (local address 127.0.0.1:40362)
0620 12:14:17.613034 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.613081 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:17.613197 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:17.614910 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:17.614914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:17.614936 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:17.615072 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.615080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.616075 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.616079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.616945 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.616956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.617829 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.617832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.617931 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.617960 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 12:14:17.618031 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 12:14:17.618161 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 12:14:17.618693 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.613533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:40285 (local address 127.0.0.1:49576)
0620 12:14:17.613886 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.613903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:17.613996 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:17.614367 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:17.614371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:17.614388 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:17.614610 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.614618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.615802 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.615805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.617262 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.617273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.618320 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.618323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.618406 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.618428 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:14:17.618503 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:14:17.618582 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":147,"thread_start_us":99,"threads_started":1}
I20260620 12:14:17.619117 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.613844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49576 (local address 127.24.141.3:40285)
0620 12:14:17.614136 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:17.614139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:17.614152 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:14:17.614199 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:17.614202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:17.614242 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:17.614329 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:17.614866 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.615667 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.617409 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.618003 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.618121 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.618186 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:17.618835 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 12:14:17.618974 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":67,"threads_started":1}
I20260620 12:14:17.619918 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.613616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.24.141.1:39599)
0620 12:14:17.614191 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:17.614196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:17.614211 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:14:17.614271 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:17.614275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:17.614324 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:17.614414 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:17.614791 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.615576 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.618275 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.618920 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.619022 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.619089 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:17.619527 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 12:14:17.619696 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":74,"threads_started":1}
I20260620 12:14:17.619869 32430 tablet_service.cc:1467] Processing CreateTablet for tablet a8d561fead434827964b8ad16c40a0dc (DEFAULT_TABLE table=test-table [id=99c13dbdafe448dbacbf6ba73091aab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:17.620235 32430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d561fead434827964b8ad16c40a0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:17.623404 32597 tablet_bootstrap.cc:492] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: Bootstrap starting.
I20260620 12:14:17.623577 32292 tablet_service.cc:1467] Processing CreateTablet for tablet a8d561fead434827964b8ad16c40a0dc (DEFAULT_TABLE table=test-table [id=99c13dbdafe448dbacbf6ba73091aab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:17.623920 32292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d561fead434827964b8ad16c40a0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:17.624620 32597 tablet_bootstrap.cc:654] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:17.626377 32597 tablet_bootstrap.cc:492] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: No bootstrap required, opened a new log
I20260620 12:14:17.626477 32597 ts_tablet_manager.cc:1397] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:14:17.626806 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.613473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:39599 (local address 127.0.0.1:45132)
0620 12:14:17.613801 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.613819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:17.613943 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:17.614441 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:17.614443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:17.614460 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:17.614656 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.614664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.615689 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.615693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.617262 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.617273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.619160 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.619164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.619344 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.619373 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 12:14:17.619467 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 12:14:17.626604 (+ 7137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":116,"thread_start_us":147,"threads_started":1}
I20260620 12:14:17.626809 32599 tablet_bootstrap.cc:492] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: Bootstrap starting.
I20260620 12:14:17.626931 32597 raft_consensus.cc:348] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.627091 32597 raft_consensus.cc:374] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:17.627133 32597 raft_consensus.cc:729] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8bbe4ed35c4b8a944802876b843431, State: Initialized, Role: FOLLOWER
I20260620 12:14:17.627280 32597 consensus_queue.cc:260] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.627588 32155 tablet_service.cc:1467] Processing CreateTablet for tablet a8d561fead434827964b8ad16c40a0dc (DEFAULT_TABLE table=test-table [id=99c13dbdafe448dbacbf6ba73091aab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:17.627730 32597 ts_tablet_manager.cc:1428] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:14:17.627925 32155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d561fead434827964b8ad16c40a0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:17.627941 32599 tablet_bootstrap.cc:654] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:17.629703 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.629358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:46461)
0620 12:14:17.629547 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.629611 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 12:14:17.630019 32599 tablet_bootstrap.cc:492] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: No bootstrap required, opened a new log
I20260620 12:14:17.630123 32599 ts_tablet_manager.cc:1397] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:14:17.630586 32599 raft_consensus.cc:348] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.630681 32599 raft_consensus.cc:374] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:17.630702 32599 raft_consensus.cc:729] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e64331a4474036b783b1ec315708c2, State: Initialized, Role: FOLLOWER
I20260620 12:14:17.630828 32599 consensus_queue.cc:260] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.630990 32602 tablet_bootstrap.cc:492] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c: Bootstrap starting.
I20260620 12:14:17.631207 32599 ts_tablet_manager.cc:1428] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:14:17.632063 32498 heartbeater.cc:503] Master 127.24.141.61:37555 was elected leader, sending a full tablet report...
I20260620 12:14:17.632313 32602 tablet_bootstrap.cc:654] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:17.632611 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.632320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:34493)
0620 12:14:17.632469 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.632527 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 12:14:17.632740 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.632213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:48679)
0620 12:14:17.632607 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.632663 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299}
I20260620 12:14:17.633889 32602 tablet_bootstrap.cc:492] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c: No bootstrap required, opened a new log
I20260620 12:14:17.633988 32602 ts_tablet_manager.cc:1397] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:14:17.634527 32602 raft_consensus.cc:348] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.634747 32602 raft_consensus.cc:374] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:17.634835 32602 raft_consensus.cc:729] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d53916f670f446b86f725d97b7fbb3c, State: Initialized, Role: FOLLOWER
I20260620 12:14:17.634851 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.633829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:44835)
0620 12:14:17.634699 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.634759 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":788}
I20260620 12:14:17.635165 32360 heartbeater.cc:503] Master 127.24.141.61:37555 was elected leader, sending a full tablet report...
I20260620 12:14:17.635162 32602 consensus_queue.cc:260] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.635617 32602 ts_tablet_manager.cc:1428] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:14:17.636426 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.636095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:57401)
0620 12:14:17.636271 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.636337 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 12:14:17.637730 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.637457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:53817)
0620 12:14:17.637593 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.637655 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 12:14:17.638048 32223 heartbeater.cc:503] Master 127.24.141.61:37555 was elected leader, sending a full tablet report...
I20260620 12:14:17.669984 32603 raft_consensus.cc:482] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:17.670189 32603 raft_consensus.cc:504] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.671660 32603 leader_election.cc:288] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599), 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3:40285)
I20260620 12:14:17.676337 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.672023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:40285 (local address 127.24.141.2:35707)
0620 12:14:17.672190 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.672208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:17.672324 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:17.672617 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:17.672621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:17.672642 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:17.672785 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.672792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.673799 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.673803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.674961 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.674973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.675786 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.675790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.675921 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.675954 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 12:14:17.676050 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 12:14:17.676152 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260620 12:14:17.676342 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.672165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:35707 (local address 127.24.141.3:40285)
0620 12:14:17.672318 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:17.672323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:17.672344 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:14:17.672416 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:17.672418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:17.672470 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:17.672580 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:17.672955 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.673672 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.675118 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.675645 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.675757 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.675803 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:17.676068 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 12:14:17.676150 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:14:17.676688 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.672180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:46021 (local address 127.24.141.1:39599)
0620 12:14:17.672352 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:17.672357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:17.672856 (+ 499us) server_negotiation.cc:407] Connection header received
0620 12:14:17.672960 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:17.672963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:17.673010 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:17.673104 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:17.673446 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.674163 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.675497 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.676062 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.676180 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.676247 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:17.676464 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 12:14:17.676552 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:14:17.676733 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.672023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:39599 (local address 127.24.141.2:46021)
0620 12:14:17.672734 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.672746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:17.672858 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:17.673141 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:17.673144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:17.673161 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:17.673309 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.673316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.674329 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.674332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.675330 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:17.675342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.676207 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.676211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:17.676312 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.676342 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:14:17.676429 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:14:17.676515 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":371,"thread_start_us":129,"threads_started":1}
I20260620 12:14:17.677306 32450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d561fead434827964b8ad16c40a0dc" candidate_uuid: "35e64331a4474036b783b1ec315708c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8bbe4ed35c4b8a944802876b843431" is_pre_election: true
I20260620 12:14:17.677436 32175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d561fead434827964b8ad16c40a0dc" candidate_uuid: "35e64331a4474036b783b1ec315708c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d53916f670f446b86f725d97b7fbb3c" is_pre_election: true
I20260620 12:14:17.677637 32175 raft_consensus.cc:2454] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e64331a4474036b783b1ec315708c2 in term 0.
I20260620 12:14:17.677642 32450 raft_consensus.cc:2454] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e64331a4474036b783b1ec315708c2 in term 0.
I20260620 12:14:17.678224 32245 leader_election.cc:302] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d53916f670f446b86f725d97b7fbb3c, 35e64331a4474036b783b1ec315708c2; no voters:
I20260620 12:14:17.678534 32603 raft_consensus.cc:2781] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:17.678596 32603 raft_consensus.cc:482] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:17.678639 32603 raft_consensus.cc:3037] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:17.679898 32603 raft_consensus.cc:504] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.680310 32603 leader_election.cc:288] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [CANDIDATE]: Term 1 election: Requested vote from peers 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599), 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3:40285)
I20260620 12:14:17.680721 32175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d561fead434827964b8ad16c40a0dc" candidate_uuid: "35e64331a4474036b783b1ec315708c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d53916f670f446b86f725d97b7fbb3c"
I20260620 12:14:17.680882 32175 raft_consensus.cc:3037] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:17.680936 32450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d561fead434827964b8ad16c40a0dc" candidate_uuid: "35e64331a4474036b783b1ec315708c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8bbe4ed35c4b8a944802876b843431"
I20260620 12:14:17.681111 32450 raft_consensus.cc:3037] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:17.682336 32175 raft_consensus.cc:2454] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e64331a4474036b783b1ec315708c2 in term 1.
I20260620 12:14:17.682428 32450 raft_consensus.cc:2454] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e64331a4474036b783b1ec315708c2 in term 1.
I20260620 12:14:17.682804 32245 leader_election.cc:302] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d53916f670f446b86f725d97b7fbb3c, 35e64331a4474036b783b1ec315708c2; no voters:
I20260620 12:14:17.683117 32603 raft_consensus.cc:2781] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:17.683390 32603 raft_consensus.cc:686] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 1 LEADER]: Becoming Leader. State: Replica: 35e64331a4474036b783b1ec315708c2, State: Running, Role: LEADER
I20260620 12:14:17.683719 32603 consensus_queue.cc:237] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } }
I20260620 12:14:17.684327 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.683986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:40393)
0620 12:14:17.684149 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.684228 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 12:14:17.687086 31859 catalog_manager.cc:5495] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e64331a4474036b783b1ec315708c2 (127.24.141.2). New cstate: current_term: 1 leader_uuid: "35e64331a4474036b783b1ec315708c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e64331a4474036b783b1ec315708c2" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 44783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 } health_report { overall_health: UNKNOWN } } }
I20260620 12:14:17.710117 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.705660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.24.141.2:44783)
0620 12:14:17.705825 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:17.705830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:17.705867 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:14:17.705967 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:17.705970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:17.706036 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:17.706132 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:17.706596 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.707164 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.708204 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:17.708477 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:17.708566 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:17.708912 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:17.709609 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:17.709705 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:17.709881 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 12:14:17.709949 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 12:14:17.714304 32175 raft_consensus.cc:1261] T a8d561fead434827964b8ad16c40a0dc P 1d53916f670f446b86f725d97b7fbb3c [term 1 FOLLOWER]: Refusing update from remote peer 35e64331a4474036b783b1ec315708c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:17.714536 32450 raft_consensus.cc:1261] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 1 FOLLOWER]: Refusing update from remote peer 35e64331a4474036b783b1ec315708c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:17.715003 32603 consensus_queue.cc:1035] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d53916f670f446b86f725d97b7fbb3c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 39599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:17.715255 32607 consensus_queue.cc:1035] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8bbe4ed35c4b8a944802876b843431" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:17.717638 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.717280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:51891)
0620 12:14:17.717434 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.717526 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 12:14:17.717695 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.717152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:60353)
0620 12:14:17.717499 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.717579 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260620 12:14:17.722892 32611 mvcc.cc:205] Tried to move back new op lower bound from 7298898565989892096 to 7298898565876228096. Current Snapshot: MvccSnapshot[applied={T|T < 7298898565989892096}]
I20260620 12:14:17.723896 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.723195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.2:49101)
0620 12:14:17.723702 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.723791 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421}
I20260620 12:14:17.726534 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.726055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.3:60265)
0620 12:14:17.726358 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.726427 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260620 12:14:17.727639 32613 mvcc.cc:205] Tried to move back new op lower bound from 7298898565989892096 to 7298898565876228096. Current Snapshot: MvccSnapshot[applied={T|T < 7298898565989892096}]
I20260620 12:14:17.728040 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.727346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:34705 (local address 127.24.141.1:34803)
0620 12:14:17.727892 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.727954 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
I20260620 12:14:17.734968 31858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:17.735186 31858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:17.740836 31858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c59ee4a783841f7b5ed08b90271dc62: Sending DeleteTablet for 3 replicas of tablet a8d561fead434827964b8ad16c40a0dc
I20260620 12:14:17.741667 32292 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d561fead434827964b8ad16c40a0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:17 UTC) from {username='oryx', principal='oryx/127.24.141.61@KRBTEST.COM'} at 127.0.0.1:40362
I20260620 12:14:17.741814 32430 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d561fead434827964b8ad16c40a0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:17 UTC) from {username='oryx', principal='oryx/127.24.141.61@KRBTEST.COM'} at 127.0.0.1:49576
I20260620 12:14:17.742101 32620 tablet_replica.cc:330] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: stopping tablet replica
I20260620 12:14:17.742244 32620 raft_consensus.cc:2229] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:17.742398 32621 tablet_replica.cc:330] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: stopping tablet replica
I20260620 12:14:17.742516 32621 raft_consensus.cc:2229] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:17.742627 32620 raft_consensus.cc:2258] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:17.742650 32621 raft_consensus.cc:2258] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:17.742599 32155 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d561fead434827964b8ad16c40a0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:17 UTC) from {username='oryx', principal='oryx/127.24.141.61@KRBTEST.COM'} at 127.0.0.1:45132
I20260620 12:14:17.743484 32621 ts_tablet_manager.cc:1905] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:17.743592 32620 ts_tablet_manager.cc:1905] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:17.743993 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 32004
I20260620 12:14:17.746534 32621 ts_tablet_manager.cc:1918] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:17.746623 32621 log.cc:1196] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/a8d561fead434827964b8ad16c40a0dc
I20260620 12:14:17.746912 32621 ts_tablet_manager.cc:1939] T a8d561fead434827964b8ad16c40a0dc P 4f8bbe4ed35c4b8a944802876b843431: Deleting consensus metadata
I20260620 12:14:17.747799 31846 catalog_manager.cc:4841] TS 4f8bbe4ed35c4b8a944802876b843431 (127.24.141.3:40285): tablet a8d561fead434827964b8ad16c40a0dc (table test-table [id=99c13dbdafe448dbacbf6ba73091aab1]) successfully deleted
I20260620 12:14:17.749339 32620 ts_tablet_manager.cc:1918] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:17.749444 32620 log.cc:1196] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/a8d561fead434827964b8ad16c40a0dc
I20260620 12:14:17.749755 32620 ts_tablet_manager.cc:1939] T a8d561fead434827964b8ad16c40a0dc P 35e64331a4474036b783b1ec315708c2: Deleting consensus metadata
I20260620 12:14:17.750594 31846 catalog_manager.cc:4841] TS 35e64331a4474036b783b1ec315708c2 (127.24.141.2:44783): tablet a8d561fead434827964b8ad16c40a0dc (table test-table [id=99c13dbdafe448dbacbf6ba73091aab1]) successfully deleted
I20260620 12:14:17.753268 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 32229
I20260620 12:14:17.754374 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:17.753462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:39599 (local address 127.0.0.1:45144)
0620 12:14:17.754162 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:17.754246 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:39599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579}
W20260620 12:14:17.754822 31844 catalog_manager.cc:4584] TS 1d53916f670f446b86f725d97b7fbb3c (127.24.141.1:39599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8d561fead434827964b8ad16c40a0dc: Network error: Client connection negotiation failed: client connection to 127.24.141.1:39599: connect: Connection refused (error 111)
I20260620 12:14:17.761849 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 32365
I20260620 12:14:17.770319 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 31721
I20260620 12:14:17.777714 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 31824
2026-06-20T12:14:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7403 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:17 dist-test-slave-rh5v krb5kdc[32625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:17 dist-test-slave-rh5v krb5kdc[32625](info): set up 2 sockets
Jun 20 12:14:17 dist-test-slave-rh5v krb5kdc[32625](info): commencing operation
krb5kdc: starting...
W20260620 12:14:18.750159 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:18 dist-test-slave-rh5v krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab.
Entry for principal oryx/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab.
I20260620 12:14:18.888818 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43977
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab
--principal=oryx/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:19.010265 32638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:19.010586 32638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:19.010654 32638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:19.015045 32638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:19.015136 32638 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:14:19.015159 32638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:19.015185 32638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:19.015205 32638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:19.015224 32638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:19.021216 32638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:43977
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.62.keytab
--principal=oryx/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:19.022444 32638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:19.023578 32638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:19.033646 32646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:19.033659 32644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:19.033659 32643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:19.033934 32638 server_base.cc:1034] running on GCE node
I20260620 12:14:19.034667 32638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:19.036156 32638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:19.037333 32638 hybrid_clock.cc:648] HybridClock initialized: now 1781957659037313 us; error 46 us; skew 500 ppm
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:19.040853 32638 init.cc:377] Logged in from keytab as oryx/127.24.141.62@KRBTEST.COM (short username oryx)
I20260620 12:14:19.042068 32638 webserver.cc:460] Webserver started at http://127.24.141.62:34833/ using document root <none> and password file <none>
I20260620 12:14:19.042723 32638 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:19.042804 32638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:19.043016 32638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:19.045876 32638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "ed215b16ce734866b48bce456d3bc6d3"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "4cb517b7687eb318e60adbb0273c7fac"
server_key_iv: "9f2bb16992b17a70df0d44fbaacc58db"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.046643 32638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "ed215b16ce734866b48bce456d3bc6d3"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "4cb517b7687eb318e60adbb0273c7fac"
server_key_iv: "9f2bb16992b17a70df0d44fbaacc58db"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.051628 32638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:14:19.054729 32653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.056380 32638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 12:14:19.056551 32638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "ed215b16ce734866b48bce456d3bc6d3"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "4cb517b7687eb318e60adbb0273c7fac"
server_key_iv: "9f2bb16992b17a70df0d44fbaacc58db"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.056680 32638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:19.077709 32638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:19.081387 32638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:19.081627 32638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:19.091053 32638 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:43977
I20260620 12:14:19.091120 32715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:43977 every 8 connection(s)
I20260620 12:14:19.092285 32638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:19.096315 32716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:19.098372 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 32638
I20260620 12:14:19.098512 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:19.098860 25140 external_mini_cluster.cc:1428] Setting key 669f3d9d42549932cc20f19a0d165586
I20260620 12:14:19.104501 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Bootstrap starting.
I20260620 12:14:19.107699 32716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:19.108642 32716 log.cc:824] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:19.111135 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: No bootstrap required, opened a new log
I20260620 12:14:19.115250 32716 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed215b16ce734866b48bce456d3bc6d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43977 } }
I20260620 12:14:19.115550 32716 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:19.115612 32716 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed215b16ce734866b48bce456d3bc6d3, State: Initialized, Role: FOLLOWER
I20260620 12:14:19.116147 32716 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed215b16ce734866b48bce456d3bc6d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43977 } }
I20260620 12:14:19.116290 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.100718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.24.141.62:43977)
0620 12:14:19.101310 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.101329 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.101371 (+ 42us) server_negotiation.cc:407] Connection header received
0620 12:14:19.102277 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.102300 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.102704 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.103038 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.104145 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.105345 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.106654 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.107265 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.107439 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.110421 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:19.110429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:19.110453 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:19.110501 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:19.112668 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.113301 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.113308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.113315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.113482 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.113797 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.113801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.113802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.114311 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:19.114570 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:19.114888 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 12:14:19.115267 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":173,"threads_started":1}
I20260620 12:14:19.116300 32716 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:19.116457 32716 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:19.116557 32716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:19.117914 32716 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed215b16ce734866b48bce456d3bc6d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43977 } }
I20260620 12:14:19.118296 32716 leader_election.cc:302] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed215b16ce734866b48bce456d3bc6d3; no voters:
I20260620 12:14:19.118680 32716 leader_election.cc:288] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:19.118844 32721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:19.119165 32721 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [term 1 LEADER]: Becoming Leader. State: Replica: ed215b16ce734866b48bce456d3bc6d3, State: Running, Role: LEADER
I20260620 12:14:19.119585 32721 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed215b16ce734866b48bce456d3bc6d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43977 } }
I20260620 12:14:19.120064 32716 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:19.121336 32722 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed215b16ce734866b48bce456d3bc6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed215b16ce734866b48bce456d3bc6d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43977 } } }
I20260620 12:14:19.121508 32722 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:19.121917 32723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed215b16ce734866b48bce456d3bc6d3. Latest consensus state: current_term: 1 leader_uuid: "ed215b16ce734866b48bce456d3bc6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed215b16ce734866b48bce456d3bc6d3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43977 } } }
I20260620 12:14:19.121999 32730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:19.122020 32723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:19.125934 32730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:19.132213 32730 catalog_manager.cc:1347] Generated new cluster ID: 22a88c4cc1a94b94af10dc5018cc4ade
I20260620 12:14:19.132318 32730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:19.159118 32730 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:19.160872 32730 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:19.168619 32730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Generated new TSK 0
I20260620 12:14:19.169577 32730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab.
Entry for principal oryx/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab.
I20260620 12:14:19.209825 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43977
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab
--principal=oryx/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:19.332270 32742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:19.332659 32742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:19.332736 32742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:19.337387 32742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:19.337493 32742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:19.337561 32742 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:14:19.337628 32742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:19.343643 32742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab
--principal=oryx/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:19.344901 32742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:19.346091 32742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:19.357494 32748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:19.357515 32750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:19.357465 32747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:19.357971 32742 server_base.cc:1034] running on GCE node
I20260620 12:14:19.358485 32742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:19.359238 32742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:19.360538 32742 hybrid_clock.cc:648] HybridClock initialized: now 1781957659360498 us; error 67 us; skew 500 ppm
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:19.364058 32742 init.cc:377] Logged in from keytab as oryx/127.24.141.1@KRBTEST.COM (short username oryx)
I20260620 12:14:19.365202 32742 webserver.cc:460] Webserver started at http://127.24.141.1:34559/ using document root <none> and password file <none>
I20260620 12:14:19.365816 32742 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:19.365885 32742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:19.366070 32742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:19.368301 32742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "55c910f97ee640ec843f22f455d2bd7a"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "a6af0db1eb050156d62d27b1962f8b30"
server_key_iv: "2d6fd55c77bb9dd6263a5ac8272091cc"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.368932 32742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "55c910f97ee640ec843f22f455d2bd7a"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "a6af0db1eb050156d62d27b1962f8b30"
server_key_iv: "2d6fd55c77bb9dd6263a5ac8272091cc"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.373865 32742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260620 12:14:19.377895 32757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.379469 32742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:14:19.379626 32742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "55c910f97ee640ec843f22f455d2bd7a"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "a6af0db1eb050156d62d27b1962f8b30"
server_key_iv: "2d6fd55c77bb9dd6263a5ac8272091cc"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.379761 32742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:19.400661 32742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:19.404160 32742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:19.404353 32742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:19.406175 32742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:19.406250 32742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.406298 32742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:19.406313 32742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.417500 32742 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:45505
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:19.418416 404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:45505 every 8 connection(s)
I20260620 12:14:19.418807 32742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:19.419266 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 32742
I20260620 12:14:19.419421 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:19.420182 25140 external_mini_cluster.cc:1428] Setting key 8c85279bc12f2b7cfc070d9bbc05a11a
I20260620 12:14:19.423593 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.407635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:33923 (local address 127.24.141.62:43977)
0620 12:14:19.407847 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.407853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.408999 (+ 1146us) server_negotiation.cc:407] Connection header received
0620 12:14:19.410536 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.410541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.410624 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.410779 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.412839 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.413547 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.415122 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.415487 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.415608 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.419169 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:19.419175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:19.419179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:19.419219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:19.421084 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.421676 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.421680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.421682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.421775 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.422293 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.422295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.422297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.422527 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:19.422632 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:19.423257 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 12:14:19.423421 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 12:14:19.424576 32767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.408147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43977 (local address 127.24.141.1:33923)
0620 12:14:19.408800 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:19.408864 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:19.410221 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:19.411036 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:19.411047 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:19.411618 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:19.412495 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.412600 (+ 105us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.413716 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.413720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.414950 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.414965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.415676 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.415681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.415909 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.416668 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:19.416691 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:19.418875 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:19.421229 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.421236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.421255 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.421560 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.421901 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.421906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.421909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.422182 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.422681 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:19.422688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:19.422999 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 12:14:19.423198 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 12:14:19.423587 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":188,"threads_started":1}
I20260620 12:14:19.426083 405 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43977
I20260620 12:14:19.426287 405 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:19.426903 405 heartbeater.cc:511] Master 127.24.141.62:43977 requested a full tablet report, sending...
I20260620 12:14:19.428645 32669 ts_manager.cc:194] Registered new tserver with Master: 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505)
I20260620 12:14:19.430207 32669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:33923
WARNING: no policy specified for oryx/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.2@KRBTEST.COM" created.
I20260620 12:14:19.439155 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.429540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39947 (local address 127.24.141.62:43977)
0620 12:14:19.429722 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.429727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.430260 (+ 533us) server_negotiation.cc:407] Connection header received
0620 12:14:19.430439 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.430443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.430501 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.430560 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.431326 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.432087 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.433304 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.433594 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.433730 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.435117 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:19.435122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:19.435126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:19.435161 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:19.437122 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.437664 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.437668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.437670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.437791 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.438175 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.438178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.438179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.438405 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:19.438504 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:19.438869 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 12:14:19.438998 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:14:19.439505 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.429514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43977 (local address 127.24.141.1:39947)
0620 12:14:19.430114 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:19.430132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:19.430298 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:19.430743 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:19.430747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:19.431022 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:19.431186 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.431196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.432269 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.432273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.433147 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.433158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.433762 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.433765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.433853 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.434459 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:19.434472 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:19.434895 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:19.437274 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.437277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.437279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.437525 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.437886 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.437889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.437891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.437992 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.438509 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:19.438512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:19.438655 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 12:14:19.438773 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 12:14:19.438978 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1,"thread_start_us":362,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab.
Entry for principal oryx/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab.
I20260620 12:14:19.453496 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43977
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab
--principal=oryx/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:19.580886 414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:19.581321 414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:19.581393 414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:19.586704 414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:19.586819 414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:19.586882 414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:14:19.586937 414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:19.593257 414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab
--principal=oryx/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:19.594571 414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:19.595925 414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:19.608752 419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:19.608815 414 server_base.cc:1034] running on GCE node
W20260620 12:14:19.608739 422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:19.608739 420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:19.609825 414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:19.610618 414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:19.611857 414 hybrid_clock.cc:648] HybridClock initialized: now 1781957659611824 us; error 50 us; skew 500 ppm
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:19.615435 414 init.cc:377] Logged in from keytab as oryx/127.24.141.2@KRBTEST.COM (short username oryx)
I20260620 12:14:19.616636 414 webserver.cc:460] Webserver started at http://127.24.141.2:41583/ using document root <none> and password file <none>
I20260620 12:14:19.617326 414 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:19.617406 414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:19.617761 414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:19.620141 414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "b7800bffff32444ba6c76f4b0462ade9"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "383f365e7c5567693e204d3a44fa6f15"
server_key_iv: "1e8962eac4e0561874ac329630da826c"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.620867 414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "b7800bffff32444ba6c76f4b0462ade9"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "383f365e7c5567693e204d3a44fa6f15"
server_key_iv: "1e8962eac4e0561874ac329630da826c"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.625861 414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:14:19.630180 429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.632107 414 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:14:19.632261 414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "b7800bffff32444ba6c76f4b0462ade9"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "383f365e7c5567693e204d3a44fa6f15"
server_key_iv: "1e8962eac4e0561874ac329630da826c"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.632414 414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:19.646173 414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:19.649742 414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:19.649940 414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:19.651727 414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:19.651795 414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.651842 414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:19.651862 414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.663406 414 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:42517
I20260620 12:14:19.663630 544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:42517 every 8 connection(s)
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:19.664542 414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:19.669706 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.652985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:33227 (local address 127.24.141.62:43977)
0620 12:14:19.653159 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.653164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.654485 (+ 1321us) server_negotiation.cc:407] Connection header received
0620 12:14:19.655934 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.655938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.656005 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.656086 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.658228 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.659040 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.660714 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.661106 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.661272 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.664839 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:19.664845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:19.664849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:19.664889 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:19.666544 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.667401 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.667405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.667406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.667530 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.668094 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.668109 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.668111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.668388 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:19.668518 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:19.669352 (+ 834us) server_negotiation.cc:299] Negotiation successful
0620 12:14:19.669533 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:14:19.670953 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.653628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43977 (local address 127.24.141.2:33227)
0620 12:14:19.654253 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:19.654332 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:19.655671 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:19.656348 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:19.656360 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:19.657030 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:19.657919 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.657946 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.659238 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.659243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.660530 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.660545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.661339 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.661344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.661610 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.662435 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:19.662459 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:19.664575 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:19.666730 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.666739 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.666759 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.667240 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.667698 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.667702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.667704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.667951 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.668546 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:19.668554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:19.668958 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 12:14:19.669250 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 12:14:19.669650 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":156,"threads_started":1}
I20260620 12:14:19.672502 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 414
I20260620 12:14:19.672662 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:19.672925 545 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43977
I20260620 12:14:19.673018 25140 external_mini_cluster.cc:1428] Setting key 12151c74567f4d43140a67106ed0453f
I20260620 12:14:19.673163 545 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:19.674033 545 heartbeater.cc:511] Master 127.24.141.62:43977 requested a full tablet report, sending...
I20260620 12:14:19.675349 32668 ts_manager.cc:194] Registered new tserver with Master: b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517)
I20260620 12:14:19.676105 32668 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:33227
WARNING: no policy specified for oryx/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:14:19.686589 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.675965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:32835 (local address 127.24.141.62:43977)
0620 12:14:19.676101 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.676106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.678738 (+ 2632us) server_negotiation.cc:407] Connection header received
0620 12:14:19.678930 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.678933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.678987 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.679055 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.679581 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.680165 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.681004 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.681328 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.681422 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.682768 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:19.682772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:19.682775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:19.682804 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:19.684544 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.685188 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.685191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.685193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.685291 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.685644 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.685648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.685650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.685910 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:19.686004 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:19.686283 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 12:14:19.686407 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:14:19.686604 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.675953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43977 (local address 127.24.141.2:32835)
0620 12:14:19.678610 (+ 2657us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:19.678627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:19.678772 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:19.679099 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:19.679102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:19.679338 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:19.679460 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.679467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.680285 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.680289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.680892 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.680901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.681446 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.681450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.681540 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.682201 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:19.682215 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:19.682591 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:19.684727 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.684732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.684734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.685002 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.685422 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.685427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.685430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.685542 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.686015 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:19.686018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:19.686121 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 12:14:19.686216 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 12:14:19.686410 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2533,"thread_start_us":281,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab.
Entry for principal oryx/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab.
I20260620 12:14:19.705024 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43977
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab
--principal=oryx/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:19.844241 554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:19.844655 554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:19.844729 554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:19.849712 554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:19.849846 554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:19.849921 554 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:14:19.849987 554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:19.856694 554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab
--principal=oryx/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.141.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:19.858323 554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:19.859750 554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:19.875398 560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:19.875386 562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:19.875379 559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:19.875975 554 server_base.cc:1034] running on GCE node
I20260620 12:14:19.876565 554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:19.877516 554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:19.878728 554 hybrid_clock.cc:648] HybridClock initialized: now 1781957659878698 us; error 50 us; skew 500 ppm
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:19.882671 554 init.cc:377] Logged in from keytab as oryx/127.24.141.3@KRBTEST.COM (short username oryx)
I20260620 12:14:19.884150 554 webserver.cc:460] Webserver started at http://127.24.141.3:39221/ using document root <none> and password file <none>
I20260620 12:14:19.884955 554 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:19.885036 554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:19.885346 554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:19.888183 554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "1a88bf18376e45daa5237c76fedace31"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "092d0e509a5b84cb01fbd56859594805"
server_key_iv: "a7eb1995e5a27007f7561298eb614aa1"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.888950 554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "1a88bf18376e45daa5237c76fedace31"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "092d0e509a5b84cb01fbd56859594805"
server_key_iv: "a7eb1995e5a27007f7561298eb614aa1"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.894817 554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:19.898887 569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.900933 554 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:14:19.901239 554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "1a88bf18376e45daa5237c76fedace31"
format_stamp: "Formatted at 2026-06-20 12:14:19 on dist-test-slave-rh5v"
server_key: "092d0e509a5b84cb01fbd56859594805"
server_key_iv: "a7eb1995e5a27007f7561298eb614aa1"
server_key_version: "encryptionkey@0"
I20260620 12:14:19.901417 554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:19.920042 554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:19.924490 554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:19.924712 554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:19.927011 554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:19.927098 554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:19.927163 554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:19.927193 554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:19.940135 554 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:44605
I20260620 12:14:19.940198 684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:44605 every 8 connection(s)
I20260620 12:14:19.941582 554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:19.944813 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 554
I20260620 12:14:19.944973 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:19.945369 25140 external_mini_cluster.cc:1428] Setting key 2307247ab071aee12bd1ff427373622f
I20260620 12:14:19.945731 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.928174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:59825 (local address 127.24.141.62:43977)
0620 12:14:19.928314 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.928318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.930084 (+ 1766us) server_negotiation.cc:407] Connection header received
0620 12:14:19.931484 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.931488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.931557 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.931637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.934189 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.934826 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.936525 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.936765 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.936870 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.940703 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:19.940709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:19.940713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:19.940760 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:19.942823 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.943617 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.943622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.943623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.943744 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.944130 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.944133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.944134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.944366 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:19.944467 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:19.945307 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 12:14:19.945554 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:14:19.946727 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.928954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43977 (local address 127.24.141.3:59825)
0620 12:14:19.929843 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:19.929919 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:19.931200 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:19.932011 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:19.932023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:19.932745 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:19.933826 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.933861 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.935054 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.935059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.936311 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.936331 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.936945 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.936949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.937211 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.938091 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:19.938140 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:19.940351 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:19.943035 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.943041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.943059 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.943444 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.943846 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.943850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.943852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.944018 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.944481 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:19.944490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:19.944886 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 12:14:19.945160 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 12:14:19.945617 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":220,"threads_started":1}
I20260620 12:14:19.948484 685 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43977
I20260620 12:14:19.948738 685 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:19.949685 685 heartbeater.cc:511] Master 127.24.141.62:43977 requested a full tablet report, sending...
I20260620 12:14:19.951207 32668 ts_manager.cc:194] Registered new tserver with Master: 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605)
I20260620 12:14:19.952085 32668 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:59825
I20260620 12:14:19.959745 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.951132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:57081 (local address 127.24.141.62:43977)
0620 12:14:19.951306 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.951312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.951656 (+ 344us) server_negotiation.cc:407] Connection header received
0620 12:14:19.951799 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.951802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.951859 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.951951 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.952577 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.953244 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.954161 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.954398 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.954496 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.955872 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:19.955875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:19.955878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:19.955902 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:19.957707 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.958234 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.958238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.958240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.958329 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:19.958605 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:19.958609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:19.958611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:19.958845 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:19.958959 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:19.959399 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 12:14:19.959559 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:14:19.959748 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.951031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43977 (local address 127.24.141.3:57081)
0620 12:14:19.951458 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:19.951478 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:19.951677 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:19.951990 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:19.951993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:19.952278 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:19.952443 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.952452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.953400 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.953404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.954032 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:19.954044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.954523 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.954526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:19.954629 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.955221 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:19.955233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:19.955708 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:19.957866 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.957870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.957872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.958121 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.958416 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:19.958427 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:19.958429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:19.958515 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:19.959021 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:19.959026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:19.959200 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 12:14:19.959330 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 12:14:19.959544 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":123,"threads_started":1}
I20260620 12:14:19.960670 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:14:19.991989 32719 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:19.983971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.24.141.62:43977)
0620 12:14:19.984219 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.984224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.984244 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:14:19.984315 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.984319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.984417 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.984558 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.985489 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.986124 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.987145 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.987400 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.987513 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:19.991809 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49060: BlockingRecv error: recv got EOF from 127.0.0.1:49060 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 12:14:19 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:20.004329 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:19.994170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.24.141.62:43977)
0620 12:14:19.994373 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:19.994379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:19.994503 (+ 124us) server_negotiation.cc:407] Connection header received
0620 12:14:19.994684 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:19.994689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:19.994764 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:19.994842 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:19.995705 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.996380 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.997380 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:19.997660 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:19.997762 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.000736 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:20.000740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:20.000743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:20.000782 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:20.002351 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:20.002907 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:20.002911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:20.002912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:20.003005 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:20.003278 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:20.003282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:20.003284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:20.003527 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:20.003662 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.003957 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.004122 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 12:14:20.009812 32670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:14:20.017539 32670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:20.047678 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.038731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.24.141.3:44605)
0620 12:14:20.039260 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.039275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.040250 (+ 975us) server_negotiation.cc:407] Connection header received
0620 12:14:20.040384 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.040390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.040675 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.040856 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:20.041622 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.042744 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.045156 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.046248 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.046371 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.046625 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.047173 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.047470 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":126,"threads_started":1}
I20260620 12:14:20.048113 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.038730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.24.141.1:45505)
0620 12:14:20.039271 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.039279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.040119 (+ 840us) server_negotiation.cc:407] Connection header received
0620 12:14:20.040246 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.040253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.040737 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.041136 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:20.041571 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.043034 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.045156 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.046248 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.046394 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.046607 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.047790 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.047947 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":95,"threads_started":1}
I20260620 12:14:20.048642 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.039987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.24.141.2:42517)
0620 12:14:20.041237 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.041248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.041278 (+ 30us) server_negotiation.cc:407] Connection header received
0620 12:14:20.041351 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.041355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.041573 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.041762 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:20.042006 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.043282 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.045251 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.046473 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.046656 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.046889 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.048101 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.048473 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":164,"threads_started":1}
I20260620 12:14:20.048679 708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.038543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:44605 (local address 127.0.0.1:58522)
0620 12:14:20.040106 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.040121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.040247 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.041372 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.041375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.041390 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:20.041490 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.041495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.042939 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.042943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.044786 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.044802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.046844 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.046852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.046979 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.047013 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.047101 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.047752 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"mutex_wait_us":596,"thread_start_us":156,"threads_started":1}
I20260620 12:14:20.049268 711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.039723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:42517 (local address 127.0.0.1:34166)
0620 12:14:20.040620 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.040637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.040754 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.041747 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.041750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.041769 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:20.041876 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.041882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.043448 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.043453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.044830 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.044841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.047073 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.047078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.047775 (+ 697us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.047802 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.047888 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.047985 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":67,"thread_start_us":125,"threads_started":1}
I20260620 12:14:20.050305 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.038543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45505 (local address 127.0.0.1:52482)
0620 12:14:20.039599 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.039678 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.039903 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.041131 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.041139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.041168 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:20.041371 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.041380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.043300 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.043303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.044789 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.044802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.046519 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.046525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.046736 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.046793 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.047118 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.048472 (+ 1354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":807,"thread_start_us":126,"threads_started":1}
I20260620 12:14:20.053429 479 tablet_service.cc:1467] Processing CreateTablet for tablet 5e53aaa96d6c49e399081245050d1401 (DEFAULT_TABLE table=test-table [id=c8b4058681684a909c662eab75496abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:20.053606 339 tablet_service.cc:1467] Processing CreateTablet for tablet 5e53aaa96d6c49e399081245050d1401 (DEFAULT_TABLE table=test-table [id=c8b4058681684a909c662eab75496abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:20.054909 479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e53aaa96d6c49e399081245050d1401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.054909 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e53aaa96d6c49e399081245050d1401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.055524 619 tablet_service.cc:1467] Processing CreateTablet for tablet 5e53aaa96d6c49e399081245050d1401 (DEFAULT_TABLE table=test-table [id=c8b4058681684a909c662eab75496abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:20.056953 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e53aaa96d6c49e399081245050d1401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.064301 713 tablet_bootstrap.cc:492] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Bootstrap starting.
I20260620 12:14:20.064954 714 tablet_bootstrap.cc:492] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Bootstrap starting.
I20260620 12:14:20.066550 715 tablet_bootstrap.cc:492] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Bootstrap starting.
I20260620 12:14:20.068647 714 tablet_bootstrap.cc:654] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.069984 714 log.cc:824] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:20.070339 713 tablet_bootstrap.cc:654] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.070425 715 tablet_bootstrap.cc:654] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.071359 713 log.cc:824] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:20.071419 715 log.cc:824] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:20.074563 713 tablet_bootstrap.cc:492] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: No bootstrap required, opened a new log
I20260620 12:14:20.074563 715 tablet_bootstrap.cc:492] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: No bootstrap required, opened a new log
I20260620 12:14:20.074563 714 tablet_bootstrap.cc:492] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: No bootstrap required, opened a new log
I20260620 12:14:20.074863 714 ts_tablet_manager.cc:1397] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 12:14:20.074863 715 ts_tablet_manager.cc:1397] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:14:20.074884 713 ts_tablet_manager.cc:1397] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 12:14:20.079900 714 raft_consensus.cc:348] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.079866 715 raft_consensus.cc:348] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.080009 713 raft_consensus.cc:348] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.080307 714 raft_consensus.cc:374] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.080307 715 raft_consensus.cc:374] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.080366 713 raft_consensus.cc:374] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.080391 714 raft_consensus.cc:729] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7800bffff32444ba6c76f4b0462ade9, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.080391 715 raft_consensus.cc:729] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a88bf18376e45daa5237c76fedace31, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.080427 713 raft_consensus.cc:729] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.081101 713 consensus_queue.cc:260] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.081249 715 consensus_queue.cc:260] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.081248 714 consensus_queue.cc:260] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.082144 405 heartbeater.cc:503] Master 127.24.141.62:43977 was elected leader, sending a full tablet report...
I20260620 12:14:20.082747 713 ts_tablet_manager.cc:1428] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260620 12:14:20.082938 714 ts_tablet_manager.cc:1428] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 12:14:20.083022 715 ts_tablet_manager.cc:1428] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260620 12:14:20.083491 685 heartbeater.cc:503] Master 127.24.141.62:43977 was elected leader, sending a full tablet report...
I20260620 12:14:20.083642 545 heartbeater.cc:503] Master 127.24.141.62:43977 was elected leader, sending a full tablet report...
I20260620 12:14:20.094293 719 raft_consensus.cc:482] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:20.094692 719 raft_consensus.cc:504] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.097003 719 leader_election.cc:288] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605), b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517)
I20260620 12:14:20.102649 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.097302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:53805 (local address 127.24.141.3:44605)
0620 12:14:20.097575 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.097581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.097662 (+ 81us) server_negotiation.cc:407] Connection header received
0620 12:14:20.097874 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.097878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.097945 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.098090 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:20.098560 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.099494 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.101190 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.101791 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.101919 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.101981 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.102389 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.102491 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 12:14:20.102898 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.098055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:40485 (local address 127.24.141.2:42517)
0620 12:14:20.098179 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.098183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.098199 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:14:20.098340 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.098342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.098383 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.098443 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:20.098746 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.099273 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.101160 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.101937 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.102102 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.102168 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.102641 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.102761 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:14:20.103370 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.097731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:42517 (local address 127.24.141.1:40485)
0620 12:14:20.098081 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.098110 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.098224 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.098476 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.098480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.098502 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:20.098643 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.098650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.099385 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.099389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.100863 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.100878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.102256 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.102261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.102401 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.102429 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.102572 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.102684 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":90,"thread_start_us":101,"threads_started":1}
I20260620 12:14:20.103484 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.097197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:44605 (local address 127.24.141.1:53805)
0620 12:14:20.097549 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.097567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.097728 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.098246 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.098250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.098276 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:20.098430 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.098437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.099628 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.099631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.100863 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.100878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.102101 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.102107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.102226 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.102260 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.102895 (+ 635us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.103013 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":37,"thread_start_us":144,"threads_started":1}
I20260620 12:14:20.104414 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e53aaa96d6c49e399081245050d1401" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9" is_pre_election: true
I20260620 12:14:20.104882 499 raft_consensus.cc:2454] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 0.
I20260620 12:14:20.105657 639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e53aaa96d6c49e399081245050d1401" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a88bf18376e45daa5237c76fedace31" is_pre_election: true
I20260620 12:14:20.105712 32760 leader_election.cc:302] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c910f97ee640ec843f22f455d2bd7a, b7800bffff32444ba6c76f4b0462ade9; no voters:
I20260620 12:14:20.106011 639 raft_consensus.cc:2454] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 0.
I20260620 12:14:20.106225 719 raft_consensus.cc:2781] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:20.106375 719 raft_consensus.cc:482] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:20.106477 719 raft_consensus.cc:3037] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.108393 719 raft_consensus.cc:504] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.109335 719 leader_election.cc:288] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 election: Requested vote from peers 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605), b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517)
I20260620 12:14:20.109804 639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e53aaa96d6c49e399081245050d1401" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a88bf18376e45daa5237c76fedace31"
I20260620 12:14:20.109997 639 raft_consensus.cc:3037] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.110014 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e53aaa96d6c49e399081245050d1401" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9"
I20260620 12:14:20.110224 499 raft_consensus.cc:3037] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.111578 639 raft_consensus.cc:2454] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 1.
I20260620 12:14:20.111831 499 raft_consensus.cc:2454] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 1.
I20260620 12:14:20.112262 32760 leader_election.cc:302] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a; no voters:
I20260620 12:14:20.112581 719 raft_consensus.cc:2781] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:20.112902 719 raft_consensus.cc:686] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 1 LEADER]: Becoming Leader. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Running, Role: LEADER
I20260620 12:14:20.113379 719 consensus_queue.cc:237] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } }
I20260620 12:14:20.117686 32669 catalog_manager.cc:5495] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1). New cstate: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } health_report { overall_health: UNKNOWN } } }
I20260620 12:14:20.149453 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.144999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.24.141.1:45505)
0620 12:14:20.145215 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.145219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.145242 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:20.145336 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.145339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.145403 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.145519 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:20.145812 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.146351 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.147469 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.147745 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.147860 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.148238 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.148913 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.149046 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.149200 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.149288 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 12:14:20.157589 499 raft_consensus.cc:1261] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Refusing update from remote peer 55c910f97ee640ec843f22f455d2bd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:20.157651 639 raft_consensus.cc:1261] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Refusing update from remote peer 55c910f97ee640ec843f22f455d2bd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:20.158645 724 consensus_queue.cc:1035] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:20.158954 719 consensus_queue.cc:1035] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:14:20.166512 546 tablet.cc:2369] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:14:20.183714 32669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:20.184041 32669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:20.188088 32669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Sending DeleteTablet for 3 replicas of tablet 5e53aaa96d6c49e399081245050d1401
I20260620 12:14:20.189462 339 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53aaa96d6c49e399081245050d1401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:20 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:52482
I20260620 12:14:20.189494 619 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53aaa96d6c49e399081245050d1401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:20 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:58522
I20260620 12:14:20.190155 737 tablet_replica.cc:330] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: stopping tablet replica
I20260620 12:14:20.190385 479 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53aaa96d6c49e399081245050d1401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:20 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:34166
I20260620 12:14:20.190559 737 raft_consensus.cc:2229] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:20.190773 32669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:14:20.190963 737 raft_consensus.cc:2258] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:20.191397 738 tablet_replica.cc:330] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: stopping tablet replica
W20260620 12:14:20.191427 32669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:20.191999 738 raft_consensus.cc:2229] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:20.192628 738 raft_consensus.cc:2258] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:20.192891 737 ts_tablet_manager.cc:1905] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:20.195469 738 ts_tablet_manager.cc:1905] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:20.196990 737 ts_tablet_manager.cc:1918] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:20.197175 737 log.cc:1196] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/5e53aaa96d6c49e399081245050d1401
I20260620 12:14:20.197505 737 ts_tablet_manager.cc:1939] T 5e53aaa96d6c49e399081245050d1401 P 1a88bf18376e45daa5237c76fedace31: Deleting consensus metadata
I20260620 12:14:20.199019 339 tablet_service.cc:1467] Processing CreateTablet for tablet a044da3aa8534c21a0475779878807aa (DEFAULT_TABLE table=test-table [id=82b50c28d1634bea9f9c1f2d7cf45481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:20.199404 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a044da3aa8534c21a0475779878807aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.199450 619 tablet_service.cc:1467] Processing CreateTablet for tablet a044da3aa8534c21a0475779878807aa (DEFAULT_TABLE table=test-table [id=82b50c28d1634bea9f9c1f2d7cf45481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:20.199824 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a044da3aa8534c21a0475779878807aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.200176 738 ts_tablet_manager.cc:1918] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:20.200340 738 log.cc:1196] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/5e53aaa96d6c49e399081245050d1401
I20260620 12:14:20.200780 738 ts_tablet_manager.cc:1939] T 5e53aaa96d6c49e399081245050d1401 P 55c910f97ee640ec843f22f455d2bd7a: Deleting consensus metadata
I20260620 12:14:20.202244 713 tablet_bootstrap.cc:492] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Bootstrap starting.
I20260620 12:14:20.203672 713 tablet_bootstrap.cc:654] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.203837 715 tablet_bootstrap.cc:492] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Bootstrap starting.
I20260620 12:14:20.199204 479 tablet_service.cc:1467] Processing CreateTablet for tablet a044da3aa8534c21a0475779878807aa (DEFAULT_TABLE table=test-table [id=82b50c28d1634bea9f9c1f2d7cf45481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:20.204471 479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a044da3aa8534c21a0475779878807aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.205168 715 tablet_bootstrap.cc:654] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.206139 32654 catalog_manager.cc:4841] TS 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505): tablet 5e53aaa96d6c49e399081245050d1401 (table test-table [id=c8b4058681684a909c662eab75496abf]) successfully deleted
I20260620 12:14:20.191516 739 tablet_replica.cc:330] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: stopping tablet replica
I20260620 12:14:20.206933 713 tablet_bootstrap.cc:492] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: No bootstrap required, opened a new log
I20260620 12:14:20.207072 713 ts_tablet_manager.cc:1397] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:14:20.207340 739 raft_consensus.cc:2229] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:20.207706 713 raft_consensus.cc:348] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.207863 713 raft_consensus.cc:374] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.207875 714 tablet_bootstrap.cc:492] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: Bootstrap starting.
I20260620 12:14:20.207901 713 raft_consensus.cc:729] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.208065 739 raft_consensus.cc:2258] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:20.208047 713 consensus_queue.cc:260] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.208328 713 ts_tablet_manager.cc:1428] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:20.209048 714 tablet_bootstrap.cc:654] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.209802 32656 catalog_manager.cc:4841] TS 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605): tablet 5e53aaa96d6c49e399081245050d1401 (table test-table [id=c8b4058681684a909c662eab75496abf]) successfully deleted
I20260620 12:14:20.210485 714 tablet_bootstrap.cc:492] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: No bootstrap required, opened a new log
I20260620 12:14:20.210605 714 ts_tablet_manager.cc:1397] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:14:20.211010 715 tablet_bootstrap.cc:492] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: No bootstrap required, opened a new log
I20260620 12:14:20.211122 715 ts_tablet_manager.cc:1397] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260620 12:14:20.211416 714 raft_consensus.cc:348] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.211545 714 raft_consensus.cc:374] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.211575 714 raft_consensus.cc:729] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7800bffff32444ba6c76f4b0462ade9, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.211681 715 raft_consensus.cc:348] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.211731 714 consensus_queue.cc:260] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.211819 715 raft_consensus.cc:374] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.211853 715 raft_consensus.cc:729] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a88bf18376e45daa5237c76fedace31, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.212021 714 ts_tablet_manager.cc:1428] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:14:20.212000 715 consensus_queue.cc:260] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.212284 715 ts_tablet_manager.cc:1428] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:14:20.214562 739 ts_tablet_manager.cc:1905] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:20.217700 739 ts_tablet_manager.cc:1918] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:20.217828 739 log.cc:1196] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/5e53aaa96d6c49e399081245050d1401
I20260620 12:14:20.218186 739 ts_tablet_manager.cc:1939] T 5e53aaa96d6c49e399081245050d1401 P b7800bffff32444ba6c76f4b0462ade9: Deleting consensus metadata
I20260620 12:14:20.219730 32656 catalog_manager.cc:4841] TS b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517): tablet 5e53aaa96d6c49e399081245050d1401 (table test-table [id=c8b4058681684a909c662eab75496abf]) successfully deleted
W20260620 12:14:20.422348 406 tablet.cc:2369] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:14:20.443192 686 tablet.cc:2369] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:14:20.490456 721 raft_consensus.cc:482] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:20.490718 721 raft_consensus.cc:504] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.492390 721 leader_election.cc:288] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505), b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517)
I20260620 12:14:20.496624 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.492834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:48709 (local address 127.24.141.1:45505)
0620 12:14:20.493162 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.493168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.493253 (+ 85us) server_negotiation.cc:407] Connection header received
0620 12:14:20.493461 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.493464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.493521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.493607 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:20.493896 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.494447 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.495578 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.496006 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.496124 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.496197 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.496357 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.496435 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260620 12:14:20.496635 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.492725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45505 (local address 127.24.141.3:48709)
0620 12:14:20.493160 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.493176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.493339 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.493630 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.493633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.493655 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:20.493796 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.493803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.494549 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.494553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.495452 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.495460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.496124 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.496128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.496241 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.496266 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.496333 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.496436 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":115,"threads_started":1}
I20260620 12:14:20.496742 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.492941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:49701 (local address 127.24.141.2:42517)
0620 12:14:20.493135 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.493140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.493518 (+ 378us) server_negotiation.cc:407] Connection header received
0620 12:14:20.493705 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.493708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.493754 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.493818 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:20.494112 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.494629 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.495734 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.496132 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.496252 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.496323 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.496497 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.496574 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:14:20.497031 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.492833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:42517 (local address 127.24.141.3:49701)
0620 12:14:20.493409 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.493423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.493555 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.493855 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.493859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.493879 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:20.494009 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.494015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.494755 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.494759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.495598 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.495609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.496268 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.496272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.496380 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.496403 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.496472 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.496577 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":114,"threads_started":1}
I20260620 12:14:20.497638 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a044da3aa8534c21a0475779878807aa" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9" is_pre_election: true
I20260620 12:14:20.497678 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a044da3aa8534c21a0475779878807aa" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c910f97ee640ec843f22f455d2bd7a" is_pre_election: true
I20260620 12:14:20.497846 499 raft_consensus.cc:2454] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 0.
I20260620 12:14:20.497982 359 raft_consensus.cc:2454] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 0.
I20260620 12:14:20.498463 570 leader_election.cc:302] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a, b7800bffff32444ba6c76f4b0462ade9; no voters:
I20260620 12:14:20.498800 721 raft_consensus.cc:2781] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:20.498865 721 raft_consensus.cc:482] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:20.498899 721 raft_consensus.cc:3037] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.500108 721 raft_consensus.cc:504] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.500528 721 leader_election.cc:288] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 election: Requested vote from peers 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505), b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517)
I20260620 12:14:20.501112 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a044da3aa8534c21a0475779878807aa" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c910f97ee640ec843f22f455d2bd7a"
I20260620 12:14:20.501197 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a044da3aa8534c21a0475779878807aa" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9"
I20260620 12:14:20.501310 359 raft_consensus.cc:3037] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.501354 499 raft_consensus.cc:3037] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.502555 499 raft_consensus.cc:2454] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 1.
I20260620 12:14:20.502578 359 raft_consensus.cc:2454] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 1.
I20260620 12:14:20.503022 572 leader_election.cc:302] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a, b7800bffff32444ba6c76f4b0462ade9; no voters:
I20260620 12:14:20.503290 721 raft_consensus.cc:2781] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:20.503541 721 raft_consensus.cc:686] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 1 LEADER]: Becoming Leader. State: Replica: 1a88bf18376e45daa5237c76fedace31, State: Running, Role: LEADER
I20260620 12:14:20.503859 721 consensus_queue.cc:237] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.507073 32670 catalog_manager.cc:5495] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a88bf18376e45daa5237c76fedace31 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "1a88bf18376e45daa5237c76fedace31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } health_report { overall_health: HEALTHY } } }
Jun 20 12:14:20 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.62@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:20.545693 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.535293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43977 (local address 127.0.0.1:49072)
0620 12:14:20.535772 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.535791 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.535960 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.536207 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.536209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.536732 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:20.536833 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.536842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.537686 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.537701 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.538354 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.538364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.538787 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.538790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.538903 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.539606 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:20.539618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:20.541570 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:20.543595 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:20.543610 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:20.543618 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:20.543961 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:20.544348 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:20.544350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:20.544351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:20.544430 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:20.544950 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:20.544958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:20.545185 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.545308 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.545501 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":33,"thread_start_us":172,"threads_started":1}
I20260620 12:14:20.545770 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.535403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.24.141.62:43977)
0620 12:14:20.535799 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.535807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.535902 (+ 95us) server_negotiation.cc:407] Connection header received
0620 12:14:20.536052 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.536055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.536094 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.536166 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:20.536971 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.537577 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.538468 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.538681 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.538772 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.541839 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:20.541844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:20.541847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:20.541881 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:20.543423 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:20.544132 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:20.544136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:20.544138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:20.544252 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:20.544541 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:20.544544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:20.544545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:20.544788 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:20.544881 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.545392 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.545584 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":341,"thread_start_us":130,"threads_started":1}
I20260620 12:14:20.551275 32670 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:49072:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:14:20.552309 32670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:20.558841 339 tablet_service.cc:1467] Processing CreateTablet for tablet 2922c23f9a594e0b8db666504ca24e81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9ad06d2c70c46fd8aa427f7c01f2909]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:14:20.558936 619 tablet_service.cc:1467] Processing CreateTablet for tablet 2922c23f9a594e0b8db666504ca24e81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9ad06d2c70c46fd8aa427f7c01f2909]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:14:20.558936 479 tablet_service.cc:1467] Processing CreateTablet for tablet 2922c23f9a594e0b8db666504ca24e81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9ad06d2c70c46fd8aa427f7c01f2909]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:14:20.559401 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2922c23f9a594e0b8db666504ca24e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.559401 479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2922c23f9a594e0b8db666504ca24e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.559413 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2922c23f9a594e0b8db666504ca24e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:20.562858 713 tablet_bootstrap.cc:492] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: Bootstrap starting.
I20260620 12:14:20.563385 714 tablet_bootstrap.cc:492] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: Bootstrap starting.
I20260620 12:14:20.563928 713 tablet_bootstrap.cc:654] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.564504 714 tablet_bootstrap.cc:654] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.565531 715 tablet_bootstrap.cc:492] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: Bootstrap starting.
I20260620 12:14:20.565893 714 tablet_bootstrap.cc:492] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: No bootstrap required, opened a new log
I20260620 12:14:20.566015 714 ts_tablet_manager.cc:1397] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 12:14:20.566485 714 raft_consensus.cc:348] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.566592 714 raft_consensus.cc:374] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.566612 714 raft_consensus.cc:729] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7800bffff32444ba6c76f4b0462ade9, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.566655 713 tablet_bootstrap.cc:492] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: No bootstrap required, opened a new log
I20260620 12:14:20.566751 713 ts_tablet_manager.cc:1397] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:14:20.566790 715 tablet_bootstrap.cc:654] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:20.566733 714 consensus_queue.cc:260] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.567040 714 ts_tablet_manager.cc:1428] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:14:20.567144 720 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.567286 720 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:14:20.567291 713 raft_consensus.cc:348] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.567449 713 raft_consensus.cc:374] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.567482 713 raft_consensus.cc:729] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.567632 713 consensus_queue.cc:260] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.567950 732 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.568109 732 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:14:20.568378 713 ts_tablet_manager.cc:1428] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:14:20.569237 715 tablet_bootstrap.cc:492] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: No bootstrap required, opened a new log
I20260620 12:14:20.569357 715 ts_tablet_manager.cc:1397] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:14:20.569967 715 raft_consensus.cc:348] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.570111 715 raft_consensus.cc:374] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:20.570149 715 raft_consensus.cc:729] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a88bf18376e45daa5237c76fedace31, State: Initialized, Role: FOLLOWER
I20260620 12:14:20.570299 715 consensus_queue.cc:260] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.570618 715 ts_tablet_manager.cc:1428] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:14:20.570566 721 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.570715 721 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:14:20.749730 732 raft_consensus.cc:482] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:20.749964 732 raft_consensus.cc:504] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.750556 732 leader_election.cc:288] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517), 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605)
I20260620 12:14:20.751302 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2922c23f9a594e0b8db666504ca24e81" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9" is_pre_election: true
I20260620 12:14:20.751302 639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2922c23f9a594e0b8db666504ca24e81" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a88bf18376e45daa5237c76fedace31" is_pre_election: true
I20260620 12:14:20.751504 499 raft_consensus.cc:2454] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 0.
I20260620 12:14:20.751518 639 raft_consensus.cc:2454] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 0.
I20260620 12:14:20.751909 32760 leader_election.cc:302] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a; no voters:
I20260620 12:14:20.752108 732 raft_consensus.cc:2781] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:20.752174 732 raft_consensus.cc:482] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:20.752210 732 raft_consensus.cc:3037] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.754099 732 raft_consensus.cc:504] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.754626 732 leader_election.cc:288] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 election: Requested vote from peers b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517), 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605)
I20260620 12:14:20.755136 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2922c23f9a594e0b8db666504ca24e81" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9"
I20260620 12:14:20.755167 639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2922c23f9a594e0b8db666504ca24e81" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a88bf18376e45daa5237c76fedace31"
I20260620 12:14:20.755313 499 raft_consensus.cc:3037] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.755319 639 raft_consensus.cc:3037] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:20.756419 639 raft_consensus.cc:2454] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 1.
I20260620 12:14:20.756580 499 raft_consensus.cc:2454] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 1.
I20260620 12:14:20.756853 32760 leader_election.cc:302] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a; no voters:
I20260620 12:14:20.757119 732 raft_consensus.cc:2781] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:20.757344 732 raft_consensus.cc:686] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [term 1 LEADER]: Becoming Leader. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Running, Role: LEADER
I20260620 12:14:20.757511 732 consensus_queue.cc:237] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:20.758253 757 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: TxnStatusTablet state changed. Reason: New leader 55c910f97ee640ec843f22f455d2bd7a. Latest consensus state: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.758461 757 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:14:20.758867 759 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:14:20.759034 759 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:14:20.759562 32670 catalog_manager.cc:5495] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1). New cstate: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } health_report { overall_health: UNKNOWN } } }
I20260620 12:14:20.811975 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.808554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.24.141.1:45505)
0620 12:14:20.808697 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.808701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.808745 (+ 44us) server_negotiation.cc:407] Connection header received
0620 12:14:20.808993 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.808996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.809045 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.809165 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:20.809443 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.809972 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.810833 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.811066 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.811157 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.811479 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.811562 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.811623 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.811742 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.811807 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 12:14:20.811975 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.808440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45505 (local address 127.0.0.1:52498)
0620 12:14:20.808654 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.808673 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.808853 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.809202 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.809205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.809213 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:14:20.809328 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.809333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.810081 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.810084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.810706 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.810716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.811177 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.811180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.811267 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.811332 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.811639 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.811656 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.811714 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.811835 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:14:20.816313 639 raft_consensus.cc:1261] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Refusing update from remote peer 55c910f97ee640ec843f22f455d2bd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:20.816489 499 raft_consensus.cc:1261] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Refusing update from remote peer 55c910f97ee640ec843f22f455d2bd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:20.816921 732 consensus_queue.cc:1035] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:20.817186 757 consensus_queue.cc:1035] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:20.819007 720 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: TxnStatusTablet state changed. Reason: New leader 55c910f97ee640ec843f22f455d2bd7a. Latest consensus state: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.819036 721 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: TxnStatusTablet state changed. Reason: New leader 55c910f97ee640ec843f22f455d2bd7a. Latest consensus state: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.819154 720 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:14:20.819154 721 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:14:20.823117 757 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.823325 757 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:14:20.823377 763 mvcc.cc:205] Tried to move back new op lower bound from 7298898578697809920 to 7298898578464231424. Current Snapshot: MvccSnapshot[applied={T|T < 7298898578697809920}]
I20260620 12:14:20.823431 757 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.823493 757 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P 55c910f97ee640ec843f22f455d2bd7a: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:14:20.823593 721 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.823709 721 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P 1a88bf18376e45daa5237c76fedace31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:14:20.826844 720 tablet_replica.cc:439] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } } }
I20260620 12:14:20.827005 720 tablet_replica.cc:442] T 2922c23f9a594e0b8db666504ca24e81 P b7800bffff32444ba6c76f4b0462ade9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:14:20.827212 764 mvcc.cc:205] Tried to move back new op lower bound from 7298898578697809920 to 7298898578464231424. Current Snapshot: MvccSnapshot[applied={T|T < 7298898578697809920}]
I20260620 12:14:20.835011 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.830997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.24.141.3:44605)
0620 12:14:20.831276 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.831281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.831297 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:14:20.831423 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.831426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.831486 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.831614 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:20.831854 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.832387 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.833340 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.833591 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.833683 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.833967 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.834473 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.834591 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.834760 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.834830 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20260620 12:14:20.848927 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.844547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45505 (local address 127.24.141.3:48749)
0620 12:14:20.844886 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.844900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.845012 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.845417 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.845420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.845429 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:14:20.845550 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.845555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.846698 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.846701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.847383 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.847393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.848045 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.848049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.848141 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.848202 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.848600 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.848605 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.848663 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.848741 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":134,"threads_started":1}
I20260620 12:14:20.848976 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.844639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:48749 (local address 127.24.141.1:45505)
0620 12:14:20.844859 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.844865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.844977 (+ 112us) server_negotiation.cc:407] Connection header received
0620 12:14:20.845158 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.845163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.845239 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.845357 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:20.845673 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.846546 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.847521 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.847878 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.848007 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.848374 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.848519 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.848597 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.848694 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.848779 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:14:20.858222 499 raft_consensus.cc:1261] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Refusing update from remote peer 1a88bf18376e45daa5237c76fedace31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:20.858280 359 raft_consensus.cc:1261] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Refusing update from remote peer 1a88bf18376e45daa5237c76fedace31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:20.858929 721 consensus_queue.cc:1035] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:20.859175 745 consensus_queue.cc:1035] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:20.889119 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.884832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:54315 (local address 127.24.141.3:44605)
0620 12:14:20.885015 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.885020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.885040 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:14:20.885169 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.885173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.885248 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.885332 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:20.885864 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.886548 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.887876 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.888121 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.888216 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.888643 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.888726 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.888783 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.888896 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.888953 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 12:14:20.889647 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.884307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:44605 (local address 127.24.141.1:54315)
0620 12:14:20.884718 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:20.884734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:20.884847 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:20.885483 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:20.885487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:20.885499 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:14:20.885683 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.885691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.886671 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.886675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.887739 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:20.887753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.888267 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.888271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:20.888383 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.888480 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:14:20.888812 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:14:20.888819 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:14:20.888880 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 12:14:20.889004 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":131,"threads_started":1}
I20260620 12:14:20.930511 32668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:20.930768 32668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:20.933894 32668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Sending DeleteTablet for 3 replicas of tablet a044da3aa8534c21a0475779878807aa
I20260620 12:14:20.935014 619 tablet_service.cc:1514] Processing DeleteTablet for tablet a044da3aa8534c21a0475779878807aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:20 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:58522
I20260620 12:14:20.935065 479 tablet_service.cc:1514] Processing DeleteTablet for tablet a044da3aa8534c21a0475779878807aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:20 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:34166
I20260620 12:14:20.935485 785 tablet_replica.cc:330] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: stopping tablet replica
I20260620 12:14:20.935660 785 raft_consensus.cc:2229] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:20.935824 785 raft_consensus.cc:2258] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:20.935998 784 tablet_replica.cc:330] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: stopping tablet replica
I20260620 12:14:20.936203 784 raft_consensus.cc:2229] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:20.936638 784 raft_consensus.cc:2258] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:20.936741 785 ts_tablet_manager.cc:1905] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:20.937702 784 ts_tablet_manager.cc:1905] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:20.939196 339 tablet_service.cc:1514] Processing DeleteTablet for tablet a044da3aa8534c21a0475779878807aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:20 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:52482
I20260620 12:14:20.940963 785 ts_tablet_manager.cc:1918] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:14:20.941087 785 log.cc:1196] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/a044da3aa8534c21a0475779878807aa
I20260620 12:14:20.941375 785 ts_tablet_manager.cc:1939] T a044da3aa8534c21a0475779878807aa P b7800bffff32444ba6c76f4b0462ade9: Deleting consensus metadata
I20260620 12:14:20.942390 32656 catalog_manager.cc:4841] TS b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517): tablet a044da3aa8534c21a0475779878807aa (table test-table [id=82b50c28d1634bea9f9c1f2d7cf45481]) successfully deleted
I20260620 12:14:20.944535 784 ts_tablet_manager.cc:1918] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:14:20.944989 784 log.cc:1196] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/a044da3aa8534c21a0475779878807aa
I20260620 12:14:20.945173 787 tablet_replica.cc:330] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: stopping tablet replica
I20260620 12:14:20.945364 787 raft_consensus.cc:2229] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:20.945515 784 ts_tablet_manager.cc:1939] T a044da3aa8534c21a0475779878807aa P 1a88bf18376e45daa5237c76fedace31: Deleting consensus metadata
I20260620 12:14:20.945528 787 raft_consensus.cc:2258] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:20.946437 787 ts_tablet_manager.cc:1905] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:20.946736 32656 catalog_manager.cc:4841] TS 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605): tablet a044da3aa8534c21a0475779878807aa (table test-table [id=82b50c28d1634bea9f9c1f2d7cf45481]) successfully deleted
I20260620 12:14:20.950160 787 ts_tablet_manager.cc:1918] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:14:20.950253 787 log.cc:1196] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/a044da3aa8534c21a0475779878807aa
I20260620 12:14:20.950500 787 ts_tablet_manager.cc:1939] T a044da3aa8534c21a0475779878807aa P 55c910f97ee640ec843f22f455d2bd7a: Deleting consensus metadata
I20260620 12:14:20.954982 32654 catalog_manager.cc:4841] TS 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505): tablet a044da3aa8534c21a0475779878807aa (table test-table [id=82b50c28d1634bea9f9c1f2d7cf45481]) successfully deleted
Jun 20 12:14:20 dist-test-slave-rh5v krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:14:20 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:20 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:14:20.984671 753 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:20.975286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.24.141.62:43977)
0620 12:14:20.975465 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.975471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.975817 (+ 346us) server_negotiation.cc:407] Connection header received
0620 12:14:20.976027 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.976031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.976098 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.976208 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:20.977045 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.977967 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.979739 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.980082 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.980215 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.984532 (+ 4317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49080: BlockingRecv error: recv got EOF from 127.0.0.1:49080 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 12:14:20 dist-test-slave-rh5v krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM
I20260620 12:14:20.998690 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:20.987112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.24.141.62:43977)
0620 12:14:20.987251 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:20.987255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:20.987683 (+ 428us) server_negotiation.cc:407] Connection header received
0620 12:14:20.987906 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:20.987911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:20.987969 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:20.988029 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:20.989005 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.989796 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.991003 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:20.991240 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:20.991334 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:20.994639 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:20.994643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:20.994646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:20.994682 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:20.996494 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:20.997046 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:20.997050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:20.997052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:20.997186 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:20.997545 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:20.997548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:20.997550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:20.997829 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:20.997940 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:20.998339 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 12:14:20.998499 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:14:21.002331 32668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:14:21.002979 32668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:21.009810 339 tablet_service.cc:1467] Processing CreateTablet for tablet 3662b704c7ae4c239f71705aa3fa5994 (DEFAULT_TABLE table=test-table [id=58389537a41040f3bdcc20a3ce944a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:21.010074 479 tablet_service.cc:1467] Processing CreateTablet for tablet 3662b704c7ae4c239f71705aa3fa5994 (DEFAULT_TABLE table=test-table [id=58389537a41040f3bdcc20a3ce944a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:21.010244 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3662b704c7ae4c239f71705aa3fa5994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:21.010339 479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3662b704c7ae4c239f71705aa3fa5994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:21.010265 619 tablet_service.cc:1467] Processing CreateTablet for tablet 3662b704c7ae4c239f71705aa3fa5994 (DEFAULT_TABLE table=test-table [id=58389537a41040f3bdcc20a3ce944a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:21.010756 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3662b704c7ae4c239f71705aa3fa5994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:21.013588 713 tablet_bootstrap.cc:492] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: Bootstrap starting.
I20260620 12:14:21.013588 714 tablet_bootstrap.cc:492] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: Bootstrap starting.
I20260620 12:14:21.014456 715 tablet_bootstrap.cc:492] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: Bootstrap starting.
I20260620 12:14:21.014698 713 tablet_bootstrap.cc:654] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:21.014698 714 tablet_bootstrap.cc:654] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:21.015647 715 tablet_bootstrap.cc:654] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:21.016976 714 tablet_bootstrap.cc:492] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: No bootstrap required, opened a new log
I20260620 12:14:21.017107 713 tablet_bootstrap.cc:492] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: No bootstrap required, opened a new log
I20260620 12:14:21.017105 714 ts_tablet_manager.cc:1397] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 12:14:21.017181 713 ts_tablet_manager.cc:1397] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:14:21.017640 714 raft_consensus.cc:348] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.017645 713 raft_consensus.cc:348] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.017777 714 raft_consensus.cc:374] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:21.017777 713 raft_consensus.cc:374] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:21.017812 714 raft_consensus.cc:729] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7800bffff32444ba6c76f4b0462ade9, State: Initialized, Role: FOLLOWER
I20260620 12:14:21.017812 713 raft_consensus.cc:729] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Initialized, Role: FOLLOWER
I20260620 12:14:21.017949 714 consensus_queue.cc:260] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.017949 713 consensus_queue.cc:260] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.018002 715 tablet_bootstrap.cc:492] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: No bootstrap required, opened a new log
I20260620 12:14:21.018106 715 ts_tablet_manager.cc:1397] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:14:21.018236 714 ts_tablet_manager.cc:1428] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:21.018235 713 ts_tablet_manager.cc:1428] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:21.018630 715 raft_consensus.cc:348] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.018776 715 raft_consensus.cc:374] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:21.018808 715 raft_consensus.cc:729] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a88bf18376e45daa5237c76fedace31, State: Initialized, Role: FOLLOWER
I20260620 12:14:21.018940 715 consensus_queue.cc:260] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.019232 715 ts_tablet_manager.cc:1428] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:14:21.301817 732 raft_consensus.cc:482] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:21.302055 732 raft_consensus.cc:504] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.302666 732 leader_election.cc:288] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517), 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605)
I20260620 12:14:21.303382 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3662b704c7ae4c239f71705aa3fa5994" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9" is_pre_election: true
I20260620 12:14:21.303452 639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3662b704c7ae4c239f71705aa3fa5994" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a88bf18376e45daa5237c76fedace31" is_pre_election: true
I20260620 12:14:21.303633 639 raft_consensus.cc:2454] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 0.
I20260620 12:14:21.303704 499 raft_consensus.cc:2454] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 0.
I20260620 12:14:21.304059 32760 leader_election.cc:302] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a; no voters:
I20260620 12:14:21.304309 732 raft_consensus.cc:2781] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:21.304395 732 raft_consensus.cc:482] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:21.304443 732 raft_consensus.cc:3037] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:21.305768 732 raft_consensus.cc:504] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.306249 732 leader_election.cc:288] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 election: Requested vote from peers b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517), 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605)
I20260620 12:14:21.306924 639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3662b704c7ae4c239f71705aa3fa5994" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a88bf18376e45daa5237c76fedace31"
I20260620 12:14:21.306986 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3662b704c7ae4c239f71705aa3fa5994" candidate_uuid: "55c910f97ee640ec843f22f455d2bd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9"
I20260620 12:14:21.307109 639 raft_consensus.cc:3037] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:21.307209 499 raft_consensus.cc:3037] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:21.308405 499 raft_consensus.cc:2454] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 1.
I20260620 12:14:21.308535 639 raft_consensus.cc:2454] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c910f97ee640ec843f22f455d2bd7a in term 1.
I20260620 12:14:21.308908 32760 leader_election.cc:302] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c910f97ee640ec843f22f455d2bd7a, b7800bffff32444ba6c76f4b0462ade9; no voters:
I20260620 12:14:21.309201 732 raft_consensus.cc:2781] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:21.309366 732 raft_consensus.cc:686] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 1 LEADER]: Becoming Leader. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Running, Role: LEADER
I20260620 12:14:21.309544 732 consensus_queue.cc:237] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.312022 32668 catalog_manager.cc:5495] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1). New cstate: current_term: 1 leader_uuid: "55c910f97ee640ec843f22f455d2bd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } health_report { overall_health: UNKNOWN } } }
I20260620 12:14:21.328260 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:21.324214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.24.141.1:45505)
0620 12:14:21.324371 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:21.324374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:21.324392 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:14:21.324523 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:21.324525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:21.324580 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:21.324674 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:21.325182 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:21.325715 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:21.326879 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:21.327152 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:21.327252 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:21.327675 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:21.327776 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:21.327867 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:21.328050 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 12:14:21.328115 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:14:21.331203 639 raft_consensus.cc:1261] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Refusing update from remote peer 55c910f97ee640ec843f22f455d2bd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:21.331398 499 raft_consensus.cc:1261] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Refusing update from remote peer 55c910f97ee640ec843f22f455d2bd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:21.331975 757 consensus_queue.cc:1035] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:21.332226 732 consensus_queue.cc:1035] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:21.342727 32668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:21.342939 32668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:21.345782 32668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Sending DeleteTablet for 3 replicas of tablet 3662b704c7ae4c239f71705aa3fa5994
I20260620 12:14:21.346695 619 tablet_service.cc:1514] Processing DeleteTablet for tablet 3662b704c7ae4c239f71705aa3fa5994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:21 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:58522
I20260620 12:14:21.346678 339 tablet_service.cc:1514] Processing DeleteTablet for tablet 3662b704c7ae4c239f71705aa3fa5994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:21 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:52482
I20260620 12:14:21.346678 479 tablet_service.cc:1514] Processing DeleteTablet for tablet 3662b704c7ae4c239f71705aa3fa5994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:21 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:34166
I20260620 12:14:21.346938 784 tablet_replica.cc:330] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: stopping tablet replica
I20260620 12:14:21.346943 785 tablet_replica.cc:330] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: stopping tablet replica
I20260620 12:14:21.346943 787 tablet_replica.cc:330] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: stopping tablet replica
I20260620 12:14:21.347074 785 raft_consensus.cc:2229] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:21.347084 787 raft_consensus.cc:2229] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:21.347095 784 raft_consensus.cc:2229] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:21.347252 785 raft_consensus.cc:2258] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:21.347273 784 raft_consensus.cc:2258] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:21.347352 787 raft_consensus.cc:2258] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:21.347528 32668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:14:21.348148 32668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:21.348506 785 ts_tablet_manager.cc:1905] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:21.348531 784 ts_tablet_manager.cc:1905] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:21.348556 787 ts_tablet_manager.cc:1905] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:21.352226 785 ts_tablet_manager.cc:1918] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:21.352350 785 log.cc:1196] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/3662b704c7ae4c239f71705aa3fa5994
I20260620 12:14:21.352393 784 ts_tablet_manager.cc:1918] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:21.352490 784 log.cc:1196] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/3662b704c7ae4c239f71705aa3fa5994
I20260620 12:14:21.352607 785 ts_tablet_manager.cc:1939] T 3662b704c7ae4c239f71705aa3fa5994 P b7800bffff32444ba6c76f4b0462ade9: Deleting consensus metadata
I20260620 12:14:21.352780 784 ts_tablet_manager.cc:1939] T 3662b704c7ae4c239f71705aa3fa5994 P 1a88bf18376e45daa5237c76fedace31: Deleting consensus metadata
I20260620 12:14:21.353233 787 ts_tablet_manager.cc:1918] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:21.353395 787 log.cc:1196] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/3662b704c7ae4c239f71705aa3fa5994
I20260620 12:14:21.353612 32656 catalog_manager.cc:4841] TS b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517): tablet 3662b704c7ae4c239f71705aa3fa5994 (table test-table [id=58389537a41040f3bdcc20a3ce944a1d]) successfully deleted
I20260620 12:14:21.353679 787 ts_tablet_manager.cc:1939] T 3662b704c7ae4c239f71705aa3fa5994 P 55c910f97ee640ec843f22f455d2bd7a: Deleting consensus metadata
I20260620 12:14:21.353835 32656 catalog_manager.cc:4841] TS 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605): tablet 3662b704c7ae4c239f71705aa3fa5994 (table test-table [id=58389537a41040f3bdcc20a3ce944a1d]) successfully deleted
I20260620 12:14:21.354523 32654 catalog_manager.cc:4841] TS 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505): tablet 3662b704c7ae4c239f71705aa3fa5994 (table test-table [id=58389537a41040f3bdcc20a3ce944a1d]) successfully deleted
I20260620 12:14:21.355240 479 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5768bb93f49959bc5a73fa77f6d11 (DEFAULT_TABLE table=test-table [id=2bb3e66b2f0646f6ab37b25fe6c90589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:21.355239 339 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5768bb93f49959bc5a73fa77f6d11 (DEFAULT_TABLE table=test-table [id=2bb3e66b2f0646f6ab37b25fe6c90589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:21.355564 479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5768bb93f49959bc5a73fa77f6d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:21.355564 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5768bb93f49959bc5a73fa77f6d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:21.357986 619 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5768bb93f49959bc5a73fa77f6d11 (DEFAULT_TABLE table=test-table [id=2bb3e66b2f0646f6ab37b25fe6c90589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:21.358376 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5768bb93f49959bc5a73fa77f6d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:21.359120 714 tablet_bootstrap.cc:492] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: Bootstrap starting.
I20260620 12:14:21.359205 713 tablet_bootstrap.cc:492] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: Bootstrap starting.
I20260620 12:14:21.360213 713 tablet_bootstrap.cc:654] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:21.360560 714 tablet_bootstrap.cc:654] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:21.361117 715 tablet_bootstrap.cc:492] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: Bootstrap starting.
I20260620 12:14:21.362277 715 tablet_bootstrap.cc:654] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:21.362776 714 tablet_bootstrap.cc:492] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: No bootstrap required, opened a new log
I20260620 12:14:21.362883 714 ts_tablet_manager.cc:1397] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 12:14:21.363488 714 raft_consensus.cc:348] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.363617 714 raft_consensus.cc:374] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:21.363639 714 raft_consensus.cc:729] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7800bffff32444ba6c76f4b0462ade9, State: Initialized, Role: FOLLOWER
I20260620 12:14:21.363837 713 tablet_bootstrap.cc:492] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: No bootstrap required, opened a new log
I20260620 12:14:21.363803 714 consensus_queue.cc:260] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.363936 713 ts_tablet_manager.cc:1397] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 12:14:21.364117 714 ts_tablet_manager.cc:1428] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:14:21.364424 713 raft_consensus.cc:348] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.364547 713 raft_consensus.cc:374] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:21.364578 713 raft_consensus.cc:729] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c910f97ee640ec843f22f455d2bd7a, State: Initialized, Role: FOLLOWER
I20260620 12:14:21.364705 713 consensus_queue.cc:260] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.364974 713 ts_tablet_manager.cc:1428] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:14:21.365726 715 tablet_bootstrap.cc:492] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: No bootstrap required, opened a new log
I20260620 12:14:21.365859 715 ts_tablet_manager.cc:1397] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 12:14:21.366334 715 raft_consensus.cc:348] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.366477 715 raft_consensus.cc:374] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:21.366523 715 raft_consensus.cc:729] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a88bf18376e45daa5237c76fedace31, State: Initialized, Role: FOLLOWER
I20260620 12:14:21.366665 715 consensus_queue.cc:260] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.367187 715 ts_tablet_manager.cc:1428] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:14:21.396185 721 raft_consensus.cc:482] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:21.396543 721 raft_consensus.cc:504] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.397259 721 leader_election.cc:288] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505), b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517)
I20260620 12:14:21.397907 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5768bb93f49959bc5a73fa77f6d11" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9" is_pre_election: true
I20260620 12:14:21.397917 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5768bb93f49959bc5a73fa77f6d11" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c910f97ee640ec843f22f455d2bd7a" is_pre_election: true
I20260620 12:14:21.398132 499 raft_consensus.cc:2454] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 0.
I20260620 12:14:21.398154 359 raft_consensus.cc:2454] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 0.
I20260620 12:14:21.398509 570 leader_election.cc:302] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a; no voters:
I20260620 12:14:21.398754 721 raft_consensus.cc:2781] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:21.398823 721 raft_consensus.cc:482] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:21.398852 721 raft_consensus.cc:3037] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:21.400398 721 raft_consensus.cc:504] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.401016 721 leader_election.cc:288] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 election: Requested vote from peers 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505), b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517)
I20260620 12:14:21.401672 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5768bb93f49959bc5a73fa77f6d11" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7800bffff32444ba6c76f4b0462ade9"
I20260620 12:14:21.401743 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5768bb93f49959bc5a73fa77f6d11" candidate_uuid: "1a88bf18376e45daa5237c76fedace31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c910f97ee640ec843f22f455d2bd7a"
I20260620 12:14:21.401873 499 raft_consensus.cc:3037] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:21.401911 359 raft_consensus.cc:3037] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:21.403160 499 raft_consensus.cc:2454] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 1.
I20260620 12:14:21.403240 359 raft_consensus.cc:2454] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a88bf18376e45daa5237c76fedace31 in term 1.
I20260620 12:14:21.403719 570 leader_election.cc:302] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a88bf18376e45daa5237c76fedace31, 55c910f97ee640ec843f22f455d2bd7a, b7800bffff32444ba6c76f4b0462ade9; no voters:
I20260620 12:14:21.403972 721 raft_consensus.cc:2781] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:21.404109 721 raft_consensus.cc:686] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 1 LEADER]: Becoming Leader. State: Replica: 1a88bf18376e45daa5237c76fedace31, State: Running, Role: LEADER
I20260620 12:14:21.404286 721 consensus_queue.cc:237] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } }
I20260620 12:14:21.406720 32669 catalog_manager.cc:5495] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a88bf18376e45daa5237c76fedace31 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "1a88bf18376e45daa5237c76fedace31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a88bf18376e45daa5237c76fedace31" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 44605 } health_report { overall_health: HEALTHY } } }
I20260620 12:14:21.435505 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:21.431391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.24.141.3:44605)
0620 12:14:21.431683 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:21.431688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:21.431703 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:14:21.431760 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:21.431762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:21.431818 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:21.431914 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:21.432200 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:21.432775 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:21.434251 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:21.434543 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:21.434647 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:21.434980 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:21.435092 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:21.435155 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:21.435258 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 12:14:21.435321 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260620 12:14:21.444572 499 raft_consensus.cc:1261] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Refusing update from remote peer 1a88bf18376e45daa5237c76fedace31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:21.444628 359 raft_consensus.cc:1261] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Refusing update from remote peer 1a88bf18376e45daa5237c76fedace31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:21.445108 745 consensus_queue.cc:1035] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c910f97ee640ec843f22f455d2bd7a" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:21.445305 721 consensus_queue.cc:1035] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7800bffff32444ba6c76f4b0462ade9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:21.449992 762 mvcc.cc:205] Tried to move back new op lower bound from 7298898581272588288 to 7298898581113638912. Current Snapshot: MvccSnapshot[applied={T|T < 7298898581272588288}]
I20260620 12:14:21.492333 32669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:21.492529 32669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:21.495364 32669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed215b16ce734866b48bce456d3bc6d3: Sending DeleteTablet for 3 replicas of tablet f4d5768bb93f49959bc5a73fa77f6d11
I20260620 12:14:21.496165 339 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d5768bb93f49959bc5a73fa77f6d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:21 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:52482
I20260620 12:14:21.496243 619 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d5768bb93f49959bc5a73fa77f6d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:21 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:58522
I20260620 12:14:21.496266 479 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d5768bb93f49959bc5a73fa77f6d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:21 UTC) from {username='oryx', principal='oryx/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:34166
I20260620 12:14:21.496448 787 tablet_replica.cc:330] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: stopping tablet replica
I20260620 12:14:21.496459 784 tablet_replica.cc:330] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: stopping tablet replica
I20260620 12:14:21.496512 785 tablet_replica.cc:330] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: stopping tablet replica
I20260620 12:14:21.496599 784 raft_consensus.cc:2229] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:21.496604 787 raft_consensus.cc:2229] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:21.496644 785 raft_consensus.cc:2229] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:21.496752 787 raft_consensus.cc:2258] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:21.496758 785 raft_consensus.cc:2258] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:21.496845 784 raft_consensus.cc:2258] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:21.497700 784 ts_tablet_manager.cc:1905] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:21.497704 787 ts_tablet_manager.cc:1905] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:21.497968 785 ts_tablet_manager.cc:1905] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:21.500576 787 ts_tablet_manager.cc:1918] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:14:21.500576 784 ts_tablet_manager.cc:1918] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:14:21.500644 785 ts_tablet_manager.cc:1918] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:14:21.500684 787 log.cc:1196] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/f4d5768bb93f49959bc5a73fa77f6d11
I20260620 12:14:21.500685 784 log.cc:1196] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/f4d5768bb93f49959bc5a73fa77f6d11
I20260620 12:14:21.500726 785 log.cc:1196] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/f4d5768bb93f49959bc5a73fa77f6d11
I20260620 12:14:21.500998 785 ts_tablet_manager.cc:1939] T f4d5768bb93f49959bc5a73fa77f6d11 P b7800bffff32444ba6c76f4b0462ade9: Deleting consensus metadata
I20260620 12:14:21.500998 784 ts_tablet_manager.cc:1939] T f4d5768bb93f49959bc5a73fa77f6d11 P 1a88bf18376e45daa5237c76fedace31: Deleting consensus metadata
I20260620 12:14:21.500998 787 ts_tablet_manager.cc:1939] T f4d5768bb93f49959bc5a73fa77f6d11 P 55c910f97ee640ec843f22f455d2bd7a: Deleting consensus metadata
I20260620 12:14:21.501977 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 32742
I20260620 12:14:21.502136 32654 catalog_manager.cc:4841] TS 55c910f97ee640ec843f22f455d2bd7a (127.24.141.1:45505): tablet f4d5768bb93f49959bc5a73fa77f6d11 (table test-table [id=2bb3e66b2f0646f6ab37b25fe6c90589]) successfully deleted
I20260620 12:14:21.502678 32656 catalog_manager.cc:4841] TS 1a88bf18376e45daa5237c76fedace31 (127.24.141.3:44605): tablet f4d5768bb93f49959bc5a73fa77f6d11 (table test-table [id=2bb3e66b2f0646f6ab37b25fe6c90589]) successfully deleted
I20260620 12:14:21.502897 32656 catalog_manager.cc:4841] TS b7800bffff32444ba6c76f4b0462ade9 (127.24.141.2:42517): tablet f4d5768bb93f49959bc5a73fa77f6d11 (table test-table [id=2bb3e66b2f0646f6ab37b25fe6c90589]) successfully deleted
I20260620 12:14:21.511107 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 414
I20260620 12:14:21.517755 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 554
I20260620 12:14:21.525345 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 32638
2026-06-20T12:14:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3747 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:21 dist-test-slave-rh5v krb5kdc[818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:21 dist-test-slave-rh5v krb5kdc[818](info): set up 2 sockets
Jun 20 12:14:21 dist-test-slave-rh5v krb5kdc[818](info): commencing operation
krb5kdc: starting...
W20260620 12:14:22.502557 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:22 dist-test-slave-rh5v krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:14:22.638253 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:36205
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:22.774407 831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:22.774788 831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:22.774853 831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:22.779471 831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:22.779587 831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:22.779616 831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:22.779639 831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:22.779659 831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:22.786128 831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:22.787696 831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:22.789089 831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:22.800169 837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:22.800050 836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:22.800209 839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:22.800371 831 server_base.cc:1034] running on GCE node
I20260620 12:14:22.801370 831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:22.802817 831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:22.804088 831 hybrid_clock.cc:648] HybridClock initialized: now 1781957662804048 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T12:14:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1293 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:22 dist-test-slave-rh5v krb5kdc[844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:22 dist-test-slave-rh5v krb5kdc[844](info): set up 2 sockets
Jun 20 12:14:22 dist-test-slave-rh5v krb5kdc[844](info): commencing operation
krb5kdc: starting...
W20260620 12:14:23.814314 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:23 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:14:23.952330 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43141
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:43141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:24.083011 858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:24.083400 858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:24.083447 858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:24.087805 858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:24.087909 858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:24.087929 858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:24.087946 858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:24.087960 858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:24.094249 858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:43141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:24.095871 858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:24.097244 858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:24.108134 864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:24.108116 866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:24.108546 858 server_base.cc:1034] running on GCE node
W20260620 12:14:24.108124 863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:24.109418 858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:24.110824 858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:24.112044 858 hybrid_clock.cc:648] HybridClock initialized: now 1781957664112024 us; error 50 us; skew 500 ppm
Jun 20 12:14:24 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:24.115891 858 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:14:24.117331 858 webserver.cc:460] Webserver started at http://127.24.141.62:45871/ using document root <none> and password file <none>
I20260620 12:14:24.118070 858 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:24.118145 858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:24.118396 858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:24.120985 858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "bd65f1db0de0477aa5b3d53e4dbbab52"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "a39392bd664b13464882897f1487b59f"
server_key_iv: "2eaaeccffe56107aac65291f8f406b36"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.121704 858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "bd65f1db0de0477aa5b3d53e4dbbab52"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "a39392bd664b13464882897f1487b59f"
server_key_iv: "2eaaeccffe56107aac65291f8f406b36"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.126658 858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:24.129977 873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.131721 858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:14:24.131871 858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "bd65f1db0de0477aa5b3d53e4dbbab52"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "a39392bd664b13464882897f1487b59f"
server_key_iv: "2eaaeccffe56107aac65291f8f406b36"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.132037 858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:24.153816 858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:24.157920 858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:24.158140 858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:24.166716 858 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:43141
I20260620 12:14:24.166803 926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:43141 every 8 connection(s)
I20260620 12:14:24.167907 858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:24.171083 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 858
I20260620 12:14:24.171222 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:24.171494 927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:24.171571 25140 external_mini_cluster.cc:1428] Setting key 89b9b8974c61396c62a8a3553ead9fb5
I20260620 12:14:24.180022 927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Bootstrap starting.
Jun 20 12:14:24 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:24.183527 927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:24.184530 927 log.cc:824] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:24.187711 927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: No bootstrap required, opened a new log
I20260620 12:14:24.189981 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:24.173387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.24.141.62:43141)
0620 12:14:24.173968 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:24.173996 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:24.174048 (+ 52us) server_negotiation.cc:407] Connection header received
0620 12:14:24.174895 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:24.174924 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:24.175445 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:24.175896 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:24.176915 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.178508 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.179623 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.180365 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.180570 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:24.183856 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:24.183865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:24.183889 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:24.183961 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:24.186216 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:24.186932 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:24.186937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:24.186943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:24.187073 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:24.187395 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:24.187398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:24.187399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:24.187965 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:24.188148 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:24.188515 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 12:14:24.188918 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":136,"threads_started":1}
I20260620 12:14:24.192078 927 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:24.192397 927 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:24.192437 927 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd65f1db0de0477aa5b3d53e4dbbab52, State: Initialized, Role: FOLLOWER
I20260620 12:14:24.192989 927 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:24.193189 927 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:24.193271 927 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:24.193397 927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:24.194485 927 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:24.194940 927 leader_election.cc:302] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd65f1db0de0477aa5b3d53e4dbbab52; no voters:
I20260620 12:14:24.195318 927 leader_election.cc:288] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:24.195523 932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:24.195823 932 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 LEADER]: Becoming Leader. State: Replica: bd65f1db0de0477aa5b3d53e4dbbab52, State: Running, Role: LEADER
I20260620 12:14:24.196301 932 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:24.196779 927 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:24.198380 934 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd65f1db0de0477aa5b3d53e4dbbab52. Latest consensus state: current_term: 1 leader_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } } }
I20260620 12:14:24.198561 934 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:24.198928 933 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } } }
I20260620 12:14:24.199018 933 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:24.199082 945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:24.203181 945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:24.210296 945 catalog_manager.cc:1347] Generated new cluster ID: da135527e2db4ab597eb687003f66586
I20260620 12:14:24.210422 945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:24.222213 945 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:24.223968 945 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:24.235517 945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Generated new TSK 0
I20260620 12:14:24.236748 945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:14:24.270612 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43141
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:24.406538 953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:24.406865 953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:24.406912 953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:24.411897 953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:24.412010 953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:24.412078 953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:24.418753 953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:24.420077 953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:24.421290 953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:24.433909 961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:24.433923 959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:24.433940 958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:24.434478 953 server_base.cc:1034] running on GCE node
I20260620 12:14:24.434949 953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:24.435647 953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:24.436856 953 hybrid_clock.cc:648] HybridClock initialized: now 1781957664436830 us; error 41 us; skew 500 ppm
Jun 20 12:14:24 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:24.440359 953 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:14:24.441639 953 webserver.cc:460] Webserver started at http://127.24.141.1:33005/ using document root <none> and password file <none>
I20260620 12:14:24.442359 953 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:24.442453 953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:24.442693 953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:24.446123 953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "7ae0b45c916e43f0a1100bf7214c1a60"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "77219f8d80d3c95e882fc0a9093bb6fc"
server_key_iv: "45d8b9fca09db971c1f07b0a7c9e09f4"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.446833 953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "7ae0b45c916e43f0a1100bf7214c1a60"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "77219f8d80d3c95e882fc0a9093bb6fc"
server_key_iv: "45d8b9fca09db971c1f07b0a7c9e09f4"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.451948 953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:14:24.455694 968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.457756 953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:24.457965 953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "7ae0b45c916e43f0a1100bf7214c1a60"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "77219f8d80d3c95e882fc0a9093bb6fc"
server_key_iv: "45d8b9fca09db971c1f07b0a7c9e09f4"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.458127 953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:24.474287 953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:24.478626 953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:24.478837 953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:24.479569 953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:24.480597 953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:24.480648 953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.480695 953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:24.480717 953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.491935 953 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:33913
I20260620 12:14:24.491976 1082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:33913 every 8 connection(s)
I20260620 12:14:24.493157 953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:24.499150 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 953
I20260620 12:14:24.499296 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:24.499626 25140 external_mini_cluster.cc:1428] Setting key 5d0bb5a7aaf9e374a205ea8323119cd6
Jun 20 12:14:24 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:14:24.512583 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:24.495360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:45209 (local address 127.24.141.62:43141)
0620 12:14:24.495677 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:24.495683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:24.496740 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 12:14:24.498540 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:24.498544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:24.498617 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:24.498706 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:24.500508 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.501313 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.503208 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.503488 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.503609 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:24.507207 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:24.507214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:24.507218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:24.507262 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:24.509489 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:24.510382 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:24.510387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:24.510389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:24.510528 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:24.511058 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:24.511061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:24.511063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:24.511352 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:24.511485 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:24.512276 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 12:14:24.512436 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 12:14:24.514222 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:24.495817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:45209)
0620 12:14:24.496513 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:24.496573 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:24.498252 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:24.498969 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:24.498981 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:24.499549 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:24.500283 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:24.500305 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.501501 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.501505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:24.502996 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:24.503012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.503694 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.503698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:24.503981 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:24.504788 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:24.504809 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:24.506871 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:24.509727 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:24.509737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:24.509759 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:24.510209 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:24.510714 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:24.510717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:24.510719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:24.510922 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:24.511520 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:24.511531 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:24.511936 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 12:14:24.512179 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 12:14:24.512743 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:14:24.516153 1083 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43141
I20260620 12:14:24.516548 1083 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:24.517261 1083 heartbeater.cc:511] Master 127.24.141.62:43141 requested a full tablet report, sending...
I20260620 12:14:24.520236 890 ts_manager.cc:194] Registered new tserver with Master: 7ae0b45c916e43f0a1100bf7214c1a60 (127.24.141.1:33913)
I20260620 12:14:24.522097 890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:45209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:14:24.538504 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43141
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:24.670243 1088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:24.670569 1088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:24.670626 1088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:24.675150 1088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:24.675247 1088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:24.675311 1088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:24.681582 1088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:24.682940 1088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:24.684276 1088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:24.696337 1096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:24.696447 1093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:24.696447 1094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:24.697105 1088 server_base.cc:1034] running on GCE node
I20260620 12:14:24.697845 1088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:24.698725 1088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:24.699992 1088 hybrid_clock.cc:648] HybridClock initialized: now 1781957664699952 us; error 64 us; skew 500 ppm
Jun 20 12:14:24 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:24.703675 1088 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:14:24.704808 1088 webserver.cc:460] Webserver started at http://127.24.141.2:33187/ using document root <none> and password file <none>
I20260620 12:14:24.705709 1088 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:24.705822 1088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:24.706086 1088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:24.708757 1088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "79f6551041184850bd801613133bfe53"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "1244f72922a84769cb96f2d207b2a227"
server_key_iv: "85e6378d4e7f19cc84dba9aba7a1a62f"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.709461 1088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "79f6551041184850bd801613133bfe53"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "1244f72922a84769cb96f2d207b2a227"
server_key_iv: "85e6378d4e7f19cc84dba9aba7a1a62f"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.714639 1088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:14:24.718197 1103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.720073 1088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260620 12:14:24.720281 1088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "79f6551041184850bd801613133bfe53"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "1244f72922a84769cb96f2d207b2a227"
server_key_iv: "85e6378d4e7f19cc84dba9aba7a1a62f"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.720425 1088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:24.743963 1088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:24.748641 1088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:24.748831 1088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:24.749538 1088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:24.750514 1088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:24.750567 1088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.750636 1088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:24.750666 1088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.761714 1088 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:40757
I20260620 12:14:24.761819 1216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:40757 every 8 connection(s)
I20260620 12:14:24.762929 1088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:24.768392 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 1088
I20260620 12:14:24.768551 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:24.768942 25140 external_mini_cluster.cc:1428] Setting key 386edd0308826d43e1bcd8f82d98880d
Jun 20 12:14:24 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:14:24.782245 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:24.765427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:38675 (local address 127.24.141.62:43141)
0620 12:14:24.765611 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:24.765616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:24.766929 (+ 1313us) server_negotiation.cc:407] Connection header received
0620 12:14:24.768533 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:24.768538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:24.768628 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:24.768775 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:24.771021 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.771865 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.773491 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.773757 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.773854 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:24.777944 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:24.777948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:24.777950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:24.777988 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:24.779574 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:24.780288 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:24.780291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:24.780292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:24.780379 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:24.780798 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:24.780800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:24.780802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:24.781093 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:24.781197 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:24.781868 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 12:14:24.782047 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:14:24.783619 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:24.765979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:38675)
0620 12:14:24.766712 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:24.766777 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:24.768201 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:24.769142 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:24.769155 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:24.769968 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:24.770804 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:24.770834 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.772068 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.772072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:24.773244 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:24.773263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:24.774039 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:24.774047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:24.774350 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:24.775387 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:24.775408 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:24.777665 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:24.779749 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:24.779758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:24.779787 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:24.780166 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:24.780507 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:24.780510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:24.780512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:24.780692 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:24.781207 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:24.781216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:24.781572 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 12:14:24.781791 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 12:14:24.782319 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":174,"threads_started":1}
I20260620 12:14:24.785499 1217 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43141
I20260620 12:14:24.785821 1217 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:24.786471 1217 heartbeater.cc:511] Master 127.24.141.62:43141 requested a full tablet report, sending...
I20260620 12:14:24.787976 890 ts_manager.cc:194] Registered new tserver with Master: 79f6551041184850bd801613133bfe53 (127.24.141.2:40757)
I20260620 12:14:24.788846 890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:38675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:14:24.800740 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43141
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:24.924165 1222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:24.924566 1222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:24.924619 1222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:24.929464 1222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:24.929587 1222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:24.929665 1222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:24.936827 1222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:24.938454 1222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:24.939857 1222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:24.953104 1230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:24.953146 1228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:24.953380 1222 server_base.cc:1034] running on GCE node
W20260620 12:14:24.953106 1227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:24.954265 1222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:24.955155 1222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:24.956370 1222 hybrid_clock.cc:648] HybridClock initialized: now 1781957664956310 us; error 93 us; skew 500 ppm
Jun 20 12:14:24 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:24.959915 1222 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:14:24.961189 1222 webserver.cc:460] Webserver started at http://127.24.141.3:43419/ using document root <none> and password file <none>
I20260620 12:14:24.961889 1222 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:24.961969 1222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:24.962186 1222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:24.964916 1222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "84995796dc934226a5ec44f9047a305f"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "8a226e1af3bf4cb3e42b6d684316c8e5"
server_key_iv: "98d053e659d2052a4f8ffe7900a3cd2b"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.965663 1222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "84995796dc934226a5ec44f9047a305f"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "8a226e1af3bf4cb3e42b6d684316c8e5"
server_key_iv: "98d053e659d2052a4f8ffe7900a3cd2b"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.972188 1222 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 12:14:24.975953 1237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:24.978026 1222 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 12:14:24.978217 1222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "84995796dc934226a5ec44f9047a305f"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "8a226e1af3bf4cb3e42b6d684316c8e5"
server_key_iv: "98d053e659d2052a4f8ffe7900a3cd2b"
server_key_version: "encryptionkey@0"
I20260620 12:14:24.978356 1222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:25.000952 1222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:25.004899 1222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:25.005165 1222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:25.005960 1222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:25.007064 1222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:25.007121 1222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.007170 1222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:25.007184 1222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.018271 1222 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:37229
I20260620 12:14:25.018293 1350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:37229 every 8 connection(s)
I20260620 12:14:25.019536 1222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:25.030200 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 1222
I20260620 12:14:25.030514 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:25.031013 25140 external_mini_cluster.cc:1428] Setting key a0084430d9956699ce014742693ce2cf
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:25.039407 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.022168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:54689 (local address 127.24.141.62:43141)
0620 12:14:25.022382 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:25.022386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:25.023695 (+ 1309us) server_negotiation.cc:407] Connection header received
0620 12:14:25.025638 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:25.025652 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:25.025742 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:25.025864 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:25.027429 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.028204 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.030707 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.031014 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.031163 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.035032 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:25.035036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:25.035039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:25.035075 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:25.036725 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:25.037532 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:25.037535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:25.037537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:25.037633 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:25.038029 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:25.038032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:25.038034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:25.038267 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:25.038361 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:25.039031 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 12:14:25.039203 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 12:14:25.040560 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.022758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:54689)
0620 12:14:25.023478 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:25.023536 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:25.025173 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:25.026137 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:25.026156 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:25.026614 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:25.027204 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.027223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.028456 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.028460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.030266 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.030297 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.031347 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.031354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.031688 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.032606 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:25.032628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:25.034815 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:25.036876 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:25.036886 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:25.036911 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:25.037406 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:25.037725 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:25.037728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:25.037730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:25.037923 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:25.038375 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:25.038386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:25.038749 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 12:14:25.038960 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 12:14:25.039429 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":176,"threads_started":1}
I20260620 12:14:25.042232 1351 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43141
I20260620 12:14:25.042668 1351 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:25.043642 1351 heartbeater.cc:511] Master 127.24.141.62:43141 requested a full tablet report, sending...
I20260620 12:14:25.045528 890 ts_manager.cc:194] Registered new tserver with Master: 84995796dc934226a5ec44f9047a305f (127.24.141.3:37229)
I20260620 12:14:25.046291 890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:54689
I20260620 12:14:25.047168 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab.
Entry for principal oryx/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab.
WARNING: no policy specified for oryx/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab.
Entry for principal oryx/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab.
WARNING: no policy specified for oryx/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.141.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab.
Entry for principal oryx/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab.
I20260620 12:14:25.133380 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 953
I20260620 12:14:25.140573 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 1088
I20260620 12:14:25.146672 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 1222
I20260620 12:14:25.153674 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 858
I20260620 12:14:25.161852 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43141
--webserver_interface=127.24.141.62
--webserver_port=45871
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:43141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:25.289559 1360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:25.289913 1360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:25.289958 1360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:25.295176 1360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:25.295291 1360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:25.295312 1360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:25.295334 1360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:25.295351 1360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:25.304857 1360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:43141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=45871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:25.306257 1360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:25.307497 1360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:25.318167 1365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:25.318167 1366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:25.318167 1368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:25.318814 1360 server_base.cc:1034] running on GCE node
I20260620 12:14:25.319731 1360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:25.321213 1360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:25.322454 1360 hybrid_clock.cc:648] HybridClock initialized: now 1781957665322423 us; error 98 us; skew 500 ppm
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957665, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:25.326015 1360 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:14:25.327656 1360 webserver.cc:460] Webserver started at http://127.24.141.62:45871/ using document root <none> and password file <none>
I20260620 12:14:25.328388 1360 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:25.328505 1360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:25.333617 1360 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:14:25.336625 1375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.338713 1360 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:25.338936 1360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "bd65f1db0de0477aa5b3d53e4dbbab52"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "a39392bd664b13464882897f1487b59f"
server_key_iv: "2eaaeccffe56107aac65291f8f406b36"
server_key_version: "encryptionkey@0"
I20260620 12:14:25.339639 1360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:25.363883 1360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:25.367977 1360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:25.368216 1360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:25.378013 1360 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:43141
I20260620 12:14:25.378043 1428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:43141 every 8 connection(s)
I20260620 12:14:25.379577 1360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:25.381757 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 1360
I20260620 12:14:25.382377 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:33913
--local_ip_for_outbound_sockets=127.24.141.1
--tserver_master_addrs=127.24.141.62:43141
--webserver_port=33005
--webserver_interface=127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:14:25.386677 1429 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:14:25.388990 1429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Bootstrap starting.
I20260620 12:14:25.411231 1429 log.cc:824] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:25.420513 1429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:14:25.421005 1429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Bootstrap complete.
I20260620 12:14:25.426206 1429 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:25.426807 1429 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd65f1db0de0477aa5b3d53e4dbbab52, State: Initialized, Role: FOLLOWER
I20260620 12:14:25.427518 1429 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:25.427668 1429 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:25.427742 1429 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:25.427860 1429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:14:25.430732 1429 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:25.431216 1429 leader_election.cc:302] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd65f1db0de0477aa5b3d53e4dbbab52; no voters:
I20260620 12:14:25.431641 1429 leader_election.cc:288] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:14:25.431797 1433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:14:25.432058 1433 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [term 2 LEADER]: Becoming Leader. State: Replica: bd65f1db0de0477aa5b3d53e4dbbab52, State: Running, Role: LEADER
I20260620 12:14:25.432440 1433 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } }
I20260620 12:14:25.433390 1429 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:25.434480 1435 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd65f1db0de0477aa5b3d53e4dbbab52. Latest consensus state: current_term: 2 leader_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } } }
I20260620 12:14:25.434443 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd65f1db0de0477aa5b3d53e4dbbab52" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43141 } } }
I20260620 12:14:25.434667 1435 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:25.434674 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:25.435176 1442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:25.440110 1442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:25.440950 1442 catalog_manager.cc:1259] Loaded cluster ID: da135527e2db4ab597eb687003f66586
I20260620 12:14:25.441013 1442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:25.443401 1442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:25.444150 1442 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bd65f1db0de0477aa5b3d53e4dbbab52: Loaded TSK: 0
I20260620 12:14:25.445504 1442 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:14:25.537955 1431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:25.538401 1431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:25.538471 1431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:25.545717 1431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:25.545850 1431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:25.545945 1431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:25.554590 1431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.1.keytab
--principal=oryx/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:33913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=33005
--tserver_master_addrs=127.24.141.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:25.556090 1431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:25.557569 1431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:25.570967 1456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:25.571007 1458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:25.570904 1455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:25.571684 1431 server_base.cc:1034] running on GCE node
I20260620 12:14:25.572295 1431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:25.573271 1431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:25.574538 1431 hybrid_clock.cc:648] HybridClock initialized: now 1781957665574493 us; error 49 us; skew 500 ppm
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957665, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:25.579201 1431 init.cc:377] Logged in from keytab as oryx/127.24.141.1@KRBTEST.COM (short username oryx)
I20260620 12:14:25.580732 1431 webserver.cc:460] Webserver started at http://127.24.141.1:33005/ using document root <none> and password file <none>
I20260620 12:14:25.581593 1431 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:25.581727 1431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:25.595131 1431 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.005s sys 0.000s
I20260620 12:14:25.598858 1465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.609172 1431 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.000s
I20260620 12:14:25.609416 1431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "7ae0b45c916e43f0a1100bf7214c1a60"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "77219f8d80d3c95e882fc0a9093bb6fc"
server_key_iv: "45d8b9fca09db971c1f07b0a7c9e09f4"
server_key_version: "encryptionkey@0"
I20260620 12:14:25.610246 1431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:25.632869 1431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:25.637847 1431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:25.638079 1431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:25.638866 1431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:25.640064 1431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:25.640136 1431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.640193 1431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:25.640224 1431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.651808 1431 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:33913
I20260620 12:14:25.651872 1580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:33913 every 8 connection(s)
I20260620 12:14:25.653012 1431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:25.656386 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 1431
I20260620 12:14:25.657038 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:40757
--local_ip_for_outbound_sockets=127.24.141.2
--tserver_master_addrs=127.24.141.62:43141
--webserver_port=33187
--webserver_interface=127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:25.674144 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.656697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:43221)
0620 12:14:25.658563 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:25.658642 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:25.660361 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:25.662145 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:25.662158 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:25.662835 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:25.663745 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.663774 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.665541 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.665545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.667214 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.667233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.668371 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.668377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.668640 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.669549 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:25.669576 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:25.673175 (+ 3599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1566,"thread_start_us":254,"threads_started":1}
W20260620 12:14:25.674883 1581 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:43141 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.141.62:43141: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
W20260620 12:14:25.675436 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:25.657447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:43221 (local address 127.24.141.62:43141)
0620 12:14:25.658305 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:25.658328 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:25.658825 (+ 497us) server_negotiation.cc:407] Connection header received
0620 12:14:25.660965 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:25.660985 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:25.661492 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:25.661892 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:25.664060 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.665328 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.667446 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.668102 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.668318 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.673600 (+ 5282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:43221: BlockingRecv error: recv got EOF from 127.24.141.1:43221 (error 108)
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":229,"threads_started":1}
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:25.683051 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.675679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:39519)
0620 12:14:25.675920 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:25.675935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:25.676053 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:25.676662 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:25.676667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:25.676859 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:25.676997 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.677003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.678271 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.678275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.678944 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.678960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.679670 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.679674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.679781 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.680370 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:25.680385 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:25.682889 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260620 12:14:25.683534 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:25.675949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39519 (local address 127.24.141.62:43141)
0620 12:14:25.676217 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:25.676222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:25.676244 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:14:25.676309 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:25.676313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:25.676386 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:25.676513 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:25.677162 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.678080 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.679115 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.679502 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.679621 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.683156 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:39519: BlockingRecv error: recv got EOF from 127.24.141.1:39519 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":206,"threads_started":1}
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:25.689278 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.683666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:60929)
0620 12:14:25.683853 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:25.683865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:25.683939 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:25.684341 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:25.684344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:25.684506 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:25.684599 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.684604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.685405 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.685409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.685948 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.685956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.686393 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.686396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.686473 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.686953 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:25.686965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:25.689118 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 12:14:25.689406 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:25.683939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:60929 (local address 127.24.141.62:43141)
0620 12:14:25.684101 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:25.684105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:25.684119 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:14:25.684165 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:25.684168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:25.684217 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:25.684300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:25.684715 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.685298 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.686064 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.686285 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.686375 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.689302 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:60929: BlockingRecv error: recv got EOF from 127.24.141.1:60929 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 12:14:25.689601 1581 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:14:25.797636 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:25.797983 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:25.798049 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:25.802615 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:25.802719 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:25.802814 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:25.808979 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.2.keytab
--principal=oryx/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:40757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=33187
--tserver_master_addrs=127.24.141.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:25.810546 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:25.811895 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:25.824012 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:25.824234 1585 server_base.cc:1034] running on GCE node
W20260620 12:14:25.823997 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:25.824005 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:25.825001 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:25.825994 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:25.827206 1585 hybrid_clock.cc:648] HybridClock initialized: now 1781957665827175 us; error 52 us; skew 500 ppm
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957665, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:25.830794 1585 init.cc:377] Logged in from keytab as oryx/127.24.141.2@KRBTEST.COM (short username oryx)
I20260620 12:14:25.832011 1585 webserver.cc:460] Webserver started at http://127.24.141.2:33187/ using document root <none> and password file <none>
I20260620 12:14:25.832702 1585 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:25.832783 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:25.838209 1585 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:25.841387 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.843302 1585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:25.843472 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "79f6551041184850bd801613133bfe53"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "1244f72922a84769cb96f2d207b2a227"
server_key_iv: "85e6378d4e7f19cc84dba9aba7a1a62f"
server_key_version: "encryptionkey@0"
I20260620 12:14:25.844213 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:25.866138 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:25.870180 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:25.870414 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:25.871254 1585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:25.872406 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:25.872460 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.872510 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:25.872524 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:25.884708 1585 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:40757
I20260620 12:14:25.884824 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:40757 every 8 connection(s)
I20260620 12:14:25.886117 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:25.887288 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 1585
I20260620 12:14:25.887768 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:37229
--local_ip_for_outbound_sockets=127.24.141.3
--tserver_master_addrs=127.24.141.62:43141
--webserver_port=43419
--webserver_interface=127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:14:25.902449 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:25.888873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:47937 (local address 127.24.141.62:43141)
0620 12:14:25.889021 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:25.889025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:25.890727 (+ 1702us) server_negotiation.cc:407] Connection header received
0620 12:14:25.892600 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:25.892605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:25.892702 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:25.892846 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:25.894512 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.895446 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.897172 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.897558 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.897716 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.902268 (+ 4552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:47937: BlockingRecv error: recv got EOF from 127.24.141.2:47937 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:14:25.903120 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.889596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:47937)
0620 12:14:25.890476 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:25.890570 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:25.892240 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:25.893102 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:25.893112 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:25.893618 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:25.894242 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.894264 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.895693 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.895696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.896947 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.896962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.897788 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.897792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.897992 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.898711 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:25.898732 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:25.901980 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":185,"threads_started":1}
W20260620 12:14:25.904057 1715 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:43141 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.141.62:43141: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:25.910507 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.904594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:43149)
0620 12:14:25.904713 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:25.904729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:25.904827 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:25.905142 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:25.905146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:25.905351 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:25.905452 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.905459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.906245 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.906249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.906822 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.906831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.907285 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.907289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.907383 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.907876 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:25.907889 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:25.910337 (+ 2448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 12:14:25.910693 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:25.904726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:43149 (local address 127.24.141.62:43141)
0620 12:14:25.904900 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:25.904905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:25.904920 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:14:25.904976 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:25.904979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:25.905023 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:25.905127 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:25.905580 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.906118 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.906944 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.907174 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.907263 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.910576 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:43149: BlockingRecv error: recv got EOF from 127.24.141.2:43149 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:25 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:25.917258 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:25.911163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:49173)
0620 12:14:25.911282 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:25.911298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:25.911381 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:25.911812 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:25.911814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:25.911961 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:25.912050 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.912056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.913168 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.913171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.913742 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:25.913753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.914315 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.914318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:25.914380 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.914993 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:25.915005 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:25.917141 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 12:14:25.917459 1715 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:14:25.917506 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:25.911312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:49173 (local address 127.24.141.62:43141)
0620 12:14:25.911492 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:25.911497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:25.911518 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:14:25.911584 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:25.911587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:25.911651 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:25.911767 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:25.912172 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.913013 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.913876 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:25.914197 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:25.914303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:25.917348 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:49173: BlockingRecv error: recv got EOF from 127.24.141.2:49173 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 12:14:26.029445 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:26.029829 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:26.029879 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:26.034533 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:26.034644 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:26.034727 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:26.041014 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/krb5kdc/oryx_127.24.141.3.keytab
--principal=oryx/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:37229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=43419
--tserver_master_addrs=127.24.141.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:26.042500 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:26.043821 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:26.055361 1726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:26.055377 1723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:26.055385 1724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:26.055655 1717 server_base.cc:1034] running on GCE node
I20260620 12:14:26.056205 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:26.056957 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:26.058164 1717 hybrid_clock.cc:648] HybridClock initialized: now 1781957666058132 us; error 52 us; skew 500 ppm
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957666, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:26.061533 1717 init.cc:377] Logged in from keytab as oryx/127.24.141.3@KRBTEST.COM (short username oryx)
I20260620 12:14:26.062661 1717 webserver.cc:460] Webserver started at http://127.24.141.3:43419/ using document root <none> and password file <none>
I20260620 12:14:26.063325 1717 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:26.063426 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:26.068311 1717 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:14:26.071173 1733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:26.072918 1717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:14:26.073117 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "84995796dc934226a5ec44f9047a305f"
format_stamp: "Formatted at 2026-06-20 12:14:24 on dist-test-slave-rh5v"
server_key: "8a226e1af3bf4cb3e42b6d684316c8e5"
server_key_iv: "98d053e659d2052a4f8ffe7900a3cd2b"
server_key_version: "encryptionkey@0"
I20260620 12:14:26.073837 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:26.095448 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:26.099403 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:26.099608 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:26.100353 1717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:26.101446 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:26.101498 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:26.101558 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:26.101583 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:26.112645 1717 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:37229
I20260620 12:14:26.112713 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:37229 every 8 connection(s)
I20260620 12:14:26.114050 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:26.118177 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 1717
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:14:26.129302 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:26.116624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:42569 (local address 127.24.141.62:43141)
0620 12:14:26.116767 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:26.116770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:26.118214 (+ 1444us) server_negotiation.cc:407] Connection header received
0620 12:14:26.120086 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:26.120091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:26.120151 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:26.120240 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:26.121923 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.122760 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.124435 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.124779 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.124915 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.129161 (+ 4246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:42569: BlockingRecv error: recv got EOF from 127.24.141.3:42569 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:14:26.130203 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:26.117243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:42569)
0620 12:14:26.118002 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:26.118073 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:26.119789 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:26.120458 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:26.120466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:26.120961 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:26.121718 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.121739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.122951 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.122954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.124271 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.124285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.125010 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.125014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.125277 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.125952 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:26.125975 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:26.128994 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":153,"threads_started":1}
I20260620 12:14:26.130703 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:26.119393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.24.141.62:43141)
0620 12:14:26.119568 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:26.119582 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:26.119641 (+ 59us) server_negotiation.cc:407] Connection header received
0620 12:14:26.119797 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:26.119801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:26.119869 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:26.119978 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:26.120901 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.121751 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.123058 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.123437 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.123597 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.125410 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:26.125418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:26.125442 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:26.125483 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:26.127555 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:26.128157 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:26.128166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:26.128174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:26.128373 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:26.128738 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:26.128744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:26.128746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:26.129379 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:26.129613 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:26.130194 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 12:14:26.130574 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 12:14:26.130784 1847 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:43141 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.141.62:43141: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:26.137933 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:26.131256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:50309)
0620 12:14:26.131358 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:26.131373 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:26.131451 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:26.132172 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:26.132175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:26.132368 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:26.132508 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.132516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.133535 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.133539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.134362 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.134374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.134910 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.134913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.134992 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.135590 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:26.135605 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:26.137792 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 12:14:26.138099 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:26.131802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:50309 (local address 127.24.141.62:43141)
0620 12:14:26.131934 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:26.131937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:26.131948 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:14:26.131981 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:26.131984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:26.132026 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:26.132102 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:26.132666 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.133418 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.134489 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.134770 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.134898 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.137964 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:50309: BlockingRecv error: recv got EOF from 127.24.141.3:50309 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:26.143556 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:26.138597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:59171)
0620 12:14:26.138720 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:26.138734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:26.138847 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:26.139095 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:26.139098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:26.139250 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:26.139329 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.139334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.140057 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.140060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.140587 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.140595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.141007 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.141011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.141112 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.141627 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:26.141638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:26.143407 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 12:14:26.143679 1584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:26.138685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:59171 (local address 127.24.141.62:43141)
0620 12:14:26.138889 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:26.138894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:26.138910 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:14:26.138953 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:26.138956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:26.138996 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:26.139062 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:26.139439 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.139959 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.140694 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.140906 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.140989 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.143565 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:59171: BlockingRecv error: recv got EOF from 127.24.141.3:59171 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 12:14:26.143914 1847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:26.696667 1851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:26.690283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:37839)
0620 12:14:26.690625 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:26.690645 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:26.690794 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:26.691107 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:26.691110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:26.691298 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:26.691410 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.691416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.692324 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.692327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.692953 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.692964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.693583 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.693586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.693657 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.694271 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:26.694285 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:26.696508 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
W20260620 12:14:26.696952 1852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:26.690434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:37839 (local address 127.24.141.62:43141)
0620 12:14:26.690719 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:26.690722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:26.690737 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:14:26.690893 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:26.690896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:26.690974 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:26.691064 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:26.691567 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.692165 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.693152 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.693461 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.693558 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.696750 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:37839: BlockingRecv error: recv got EOF from 127.24.141.1:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:26 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:26.925227 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:26.918265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:46833)
0620 12:14:26.918644 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:26.918666 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:26.918851 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:26.919150 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:26.919154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:26.919415 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:26.919540 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.919548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.920413 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.920417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.921037 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:26.921048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.921569 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.921574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:26.921662 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.922291 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:26.922306 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:26.924960 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":152,"threads_started":1}
W20260620 12:14:26.925387 1852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:26.918507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:46833 (local address 127.24.141.62:43141)
0620 12:14:26.918697 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:26.918705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:26.918776 (+ 71us) server_negotiation.cc:407] Connection header received
0620 12:14:26.918975 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:26.918978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:26.919033 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:26.919102 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:26.919663 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.920252 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.921212 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:26.921451 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:26.921544 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:26.925255 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:46833: BlockingRecv error: recv got EOF from 127.24.141.2:46833 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 12:14:27 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:27 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:27.152467 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:27.144859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:49785)
0620 12:14:27.145245 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:27.145266 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:27.145499 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:27.145815 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:27.145818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:27.146059 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:27.146232 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:27.146241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.147149 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.147153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:27.147804 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:27.147816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.148374 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.148378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:27.148493 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:27.149636 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:27.149653 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:27.152256 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":96,"threads_started":1}
W20260620 12:14:27.152752 1852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:27.144984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:49785 (local address 127.24.141.62:43141)
0620 12:14:27.145210 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:27.145215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:27.145387 (+ 172us) server_negotiation.cc:407] Connection header received
0620 12:14:27.145639 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:27.145642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:27.145698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:27.145770 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:27.146418 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.147022 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.147962 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.148247 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.148348 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:27.152581 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:49785: BlockingRecv error: recv got EOF from 127.24.141.3:49785 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 12:14:27 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:27 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:27.704275 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:27.697816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:46131)
0620 12:14:27.698163 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:27.698184 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:27.698330 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:27.698666 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:27.698669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:27.698935 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:27.699087 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:27.699095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.699998 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.700001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:27.700638 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:27.700647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.701197 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.701200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:27.701269 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:27.701876 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:27.701889 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:27.704084 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
W20260620 12:14:27.704487 1856 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:27.697948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:46131 (local address 127.24.141.62:43141)
0620 12:14:27.698240 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:27.698245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:27.698265 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:14:27.698447 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:27.698450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:27.698512 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:27.698614 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:27.699253 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.699863 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.700797 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.701044 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.701184 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:27.704310 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:46131: BlockingRecv error: recv got EOF from 127.24.141.1:46131 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
Jun 20 12:14:27 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:27 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:27.937131 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:27.926319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:45275)
0620 12:14:27.926657 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:27.926672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:27.926799 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:27.927085 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:27.927087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:27.927296 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:27.927427 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:27.927434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.928198 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.928202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:27.928854 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:27.928865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.929429 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.929432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:27.929522 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:27.934234 (+ 4712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:27.934253 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:27.936914 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
W20260620 12:14:27.937325 1856 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:27.926404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:45275 (local address 127.24.141.62:43141)
0620 12:14:27.926533 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:27.926536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:27.926754 (+ 218us) server_negotiation.cc:407] Connection header received
0620 12:14:27.926903 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:27.926905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:27.926954 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:27.927044 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:27.927530 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.928066 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.929004 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:27.929279 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:27.929372 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:27.937176 (+ 7804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:45275: BlockingRecv error: recv got EOF from 127.24.141.2:45275 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 12:14:28 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:28 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:28.160346 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:28.153807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:35023)
0620 12:14:28.154123 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:28.154138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:28.154246 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:28.154606 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:28.154608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:28.154814 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:28.154936 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:28.154942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.155875 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.155880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:28.156632 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:28.156649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.157281 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.157284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:28.157378 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:28.158020 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:28.158034 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:28.160187 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":113,"threads_started":1}
W20260620 12:14:28.160568 1856 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:28.154089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:35023 (local address 127.24.141.62:43141)
0620 12:14:28.154306 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:28.154310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:28.154331 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:14:28.154388 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:28.154390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:28.154456 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:28.154568 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:28.155106 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.155718 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.156792 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.157137 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.157270 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:28.160388 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:35023: BlockingRecv error: recv got EOF from 127.24.141.3:35023 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 20 12:14:28 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:28 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:28.712623 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:28.705449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:40023)
0620 12:14:28.705842 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:28.705862 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:28.706036 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:28.706367 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:28.706369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:28.706645 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:28.706794 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:28.706800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.707845 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.707850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:28.708690 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:28.708706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.709464 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.709467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:28.709564 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:28.710241 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:28.710255 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:28.712460 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
W20260620 12:14:28.712857 1860 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:28.705619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:40023 (local address 127.24.141.62:43141)
0620 12:14:28.705935 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:28.705939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:28.705966 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:14:28.706150 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:28.706153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:28.706207 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:28.706313 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:28.706967 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.707620 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.708905 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.709303 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.709437 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:28.712693 (+ 3256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:40023: BlockingRecv error: recv got EOF from 127.24.141.1:40023 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
Jun 20 12:14:28 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:28 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:28.944578 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:28.938109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:46233)
0620 12:14:28.938571 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:28.938590 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:28.938732 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:28.939100 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:28.939104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:28.939377 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:28.939513 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:28.939524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.940435 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.940439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:28.941104 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:28.941116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.941645 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.941648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:28.941760 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:28.942350 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:28.942362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:28.944379 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":122,"threads_started":1}
W20260620 12:14:28.944880 1860 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:28.938289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:46233 (local address 127.24.141.62:43141)
0620 12:14:28.938543 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:28.938549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:28.938671 (+ 122us) server_negotiation.cc:407] Connection header received
0620 12:14:28.938862 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:28.938865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:28.938928 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:28.939023 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:28.939703 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.940276 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.941247 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:28.941495 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:28.941584 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:28.944601 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:46233: BlockingRecv error: recv got EOF from 127.24.141.2:46233 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 12:14:29 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:29 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:29.167414 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:29.161212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:35221)
0620 12:14:29.161553 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:29.161572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:29.161722 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:29.162005 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:29.162008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:29.162242 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:29.162353 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:29.162359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.163153 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.163157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:29.163769 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:29.163779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.164231 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.164235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:29.164317 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:29.164947 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:29.164962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:29.167244 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":91,"threads_started":1}
W20260620 12:14:29.167599 1860 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:29.161337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:35221 (local address 127.24.141.62:43141)
0620 12:14:29.161535 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:29.161541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:29.161658 (+ 117us) server_negotiation.cc:407] Connection header received
0620 12:14:29.161825 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:29.161828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:29.161888 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:29.161962 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:29.162475 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.163044 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.163899 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.164114 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.164215 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:29.167430 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:35221: BlockingRecv error: recv got EOF from 127.24.141.3:35221 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 12:14:29 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:29 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:29.720227 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:29.713629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:57029)
0620 12:14:29.714014 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:29.714037 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:29.714181 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:29.714566 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:29.714570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:29.714827 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:29.714954 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:29.714960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.716083 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.716085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:29.716781 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:29.716791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.717300 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.717302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:29.717374 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:29.717917 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:29.717928 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:29.720078 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":93,"threads_started":1}
W20260620 12:14:29.720558 1864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:29.713861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:57029 (local address 127.24.141.62:43141)
0620 12:14:29.714207 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:29.714212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:29.714253 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:14:29.714311 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:29.714314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:29.714379 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:29.714485 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:29.715130 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.715941 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.716896 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.717178 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.717283 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:29.720368 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:57029: BlockingRecv error: recv got EOF from 127.24.141.1:57029 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":108,"threads_started":1}
Jun 20 12:14:29 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:29 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:29.952739 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:29.945709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:58063)
0620 12:14:29.946024 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:29.946039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:29.946169 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:29.946597 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:29.946600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:29.946808 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:29.946935 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:29.946942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.947862 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.947866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:29.948517 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:29.948530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.949194 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.949198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:29.949280 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:29.949843 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:29.949859 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:29.952582 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
W20260620 12:14:29.952847 1864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:29.945886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:58063 (local address 127.24.141.62:43141)
0620 12:14:29.946046 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:29.946050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:29.946120 (+ 70us) server_negotiation.cc:407] Connection header received
0620 12:14:29.946340 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:29.946344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:29.946422 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:29.946530 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:29.947074 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.947702 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.948742 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:29.948998 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:29.949173 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:29.952712 (+ 3539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:58063: BlockingRecv error: recv got EOF from 127.24.141.2:58063 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 12:14:30 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:30 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:30.175724 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:30.168702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:54347)
0620 12:14:30.169130 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:30.169150 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:30.169325 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:30.169654 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:30.169658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:30.169949 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:30.170119 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:30.170128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.171004 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.171007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:30.171674 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:30.171685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.172216 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.172218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:30.172285 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:30.172935 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:30.172948 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:30.175565 (+ 2617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
W20260620 12:14:30.176128 1864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:30.168819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:54347 (local address 127.24.141.62:43141)
0620 12:14:30.168985 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:30.168989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:30.169295 (+ 306us) server_negotiation.cc:407] Connection header received
0620 12:14:30.169427 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:30.169430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:30.169501 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:30.169611 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:30.170254 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.170842 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.171837 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.172099 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.172201 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:30.175814 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:54347: BlockingRecv error: recv got EOF from 127.24.141.3:54347 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 12:14:30 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:30 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:30.727655 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:30.721276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:38451)
0620 12:14:30.721585 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:30.721601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:30.721704 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:30.722032 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:30.722035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:30.722255 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:30.722377 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:30.722382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.723279 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.723283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:30.724080 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:30.724091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.724627 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.724631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:30.724716 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:30.725395 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:30.725411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:30.727490 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
W20260620 12:14:30.727869 1868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:30.721460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:38451 (local address 127.24.141.62:43141)
0620 12:14:30.721761 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:30.721764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:30.721781 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:14:30.721833 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:30.721836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:30.721882 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:30.721997 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:30.722540 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.723135 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.724198 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.724487 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.724598 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:30.727702 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:38451: BlockingRecv error: recv got EOF from 127.24.141.1:38451 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
Jun 20 12:14:30 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:30 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:30.964130 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:30.953768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:33849)
0620 12:14:30.954284 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:30.954302 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:30.954434 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:30.954839 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:30.954842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:30.955039 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:30.955160 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:30.955165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.955993 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.955996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:30.956665 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:30.956677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.957341 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.957345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:30.957559 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:30.961245 (+ 3686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:30.961268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:30.963980 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":132,"threads_started":1}
W20260620 12:14:30.964434 1868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:30.953928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:33849 (local address 127.24.141.62:43141)
0620 12:14:30.954101 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:30.954106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:30.954391 (+ 285us) server_negotiation.cc:407] Connection header received
0620 12:14:30.954567 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:30.954571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:30.954662 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:30.954763 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:30.955275 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.955867 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.956813 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:30.957150 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:30.957257 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:30.964282 (+ 7025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:33849: BlockingRecv error: recv got EOF from 127.24.141.2:33849 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 12:14:31 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:31 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:31.183277 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:31.176799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:49535)
0620 12:14:31.177220 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:31.177245 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:31.177455 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:31.177765 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:31.177769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:31.178017 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:31.178147 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:31.178155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.178971 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.178975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:31.179701 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:31.179716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.180200 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.180202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:31.180268 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:31.180865 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:31.180882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:31.183116 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
W20260620 12:14:31.183487 1868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:31.176927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:49535 (local address 127.24.141.62:43141)
0620 12:14:31.177123 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:31.177128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:31.177372 (+ 244us) server_negotiation.cc:407] Connection header received
0620 12:14:31.177574 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:31.177577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:31.177641 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:31.177727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:31.178287 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.178833 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.179848 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.180079 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.180186 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:31.183336 (+ 3150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:49535: BlockingRecv error: recv got EOF from 127.24.141.3:49535 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 12:14:31 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:31 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:31.735672 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:31.728677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:39055)
0620 12:14:31.728995 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:31.729011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:31.729165 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:31.729704 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:31.729707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:31.729926 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:31.730054 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:31.730060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.731161 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.731167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:31.731801 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:31.731814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.732350 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.732354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:31.732446 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:31.733121 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:31.733141 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:31.735501 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
W20260620 12:14:31.735965 1872 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:31.728812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39055 (local address 127.24.141.62:43141)
0620 12:14:31.729101 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:31.729105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:31.729128 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:31.729349 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:31.729353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:31.729516 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:31.729641 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:31.730240 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.730963 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.731954 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.732213 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.732321 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:31.735691 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:39055: BlockingRecv error: recv got EOF from 127.24.141.1:39055 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
Jun 20 12:14:31 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:31 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:31.971508 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:31.965275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:60303)
0620 12:14:31.965631 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:31.965648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:31.965790 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:31.966099 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:31.966101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:31.966326 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:31.966456 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:31.966462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.967314 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.967318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:31.967989 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:31.967999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.968531 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.968534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:31.968615 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:31.969243 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:31.969255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:31.971345 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
W20260620 12:14:31.971710 1872 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:31.965321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:60303 (local address 127.24.141.62:43141)
0620 12:14:31.965715 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:31.965718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:31.965740 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:14:31.965913 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:31.965916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:31.965973 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:31.966063 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:31.966597 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.967171 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.968130 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:31.968372 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:31.968467 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:31.971571 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:60303: BlockingRecv error: recv got EOF from 127.24.141.2:60303 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 12:14:32 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:32 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:32.190711 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:32.184479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:59377)
0620 12:14:32.184818 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:32.184834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:32.184953 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:32.185314 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:32.185316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:32.185516 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:32.185660 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:32.185666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.186576 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.186579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:32.187181 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:32.187191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.187767 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.187771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:32.187873 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:32.188427 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:32.188440 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:32.190557 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":86,"threads_started":1}
W20260620 12:14:32.190898 1872 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:32.184636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:59377 (local address 127.24.141.62:43141)
0620 12:14:32.184793 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:32.184797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:32.184902 (+ 105us) server_negotiation.cc:407] Connection header received
0620 12:14:32.185046 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:32.185049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:32.185140 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:32.185254 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:32.185783 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.186410 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.187333 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.187605 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.187697 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:32.190750 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:59377: BlockingRecv error: recv got EOF from 127.24.141.3:59377 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 12:14:32 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:32 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:32.743837 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:32.736722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:56871)
0620 12:14:32.737113 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:32.737130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:32.737259 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:32.737567 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:32.737570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:32.737786 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:32.737910 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:32.737915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.738858 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.738860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:32.739569 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:32.739581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.740240 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.740245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:32.740349 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:32.741020 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:32.741035 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:32.743556 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":93,"threads_started":1}
W20260620 12:14:32.743973 1876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:32.736869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:56871 (local address 127.24.141.62:43141)
0620 12:14:32.737242 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:32.737246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:32.737269 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:32.737367 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:32.737370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:32.737419 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:32.737510 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:32.738080 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.738745 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.739766 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.740082 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.740214 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:32.743816 (+ 3602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:56871: BlockingRecv error: recv got EOF from 127.24.141.1:56871 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
Jun 20 12:14:32 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:32 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:32.982511 1877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:32.972679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:35895)
0620 12:14:32.973169 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:32.973187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:32.973404 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:32.973698 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:32.973701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:32.973949 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:32.974058 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:32.974064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.975073 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.975079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:32.975896 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:32.975911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.976548 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.976552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:32.976656 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:32.980310 (+ 3654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:32.980326 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:32.982344 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":117,"threads_started":1}
W20260620 12:14:32.982797 1876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:32.972849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:35895 (local address 127.24.141.62:43141)
0620 12:14:32.973122 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:32.973127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:32.973294 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:14:32.973525 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:32.973529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:32.973583 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:32.973654 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:32.974179 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.974814 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.976053 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:32.976374 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:32.976481 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:32.982628 (+ 6147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:35895: BlockingRecv error: recv got EOF from 127.24.141.2:35895 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 20 12:14:33 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:33 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:33.198613 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:33.191787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:34807)
0620 12:14:33.192221 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:33.192238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:33.192380 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:33.192663 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:33.192666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:33.192895 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:33.193021 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:33.193027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.193940 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.193943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:33.194607 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:33.194618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.195076 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.195078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:33.195154 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:33.195771 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:33.195785 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:33.198441 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":166,"threads_started":1}
W20260620 12:14:33.198798 1876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:33.191892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:34807 (local address 127.24.141.62:43141)
0620 12:14:33.192124 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:33.192129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:33.192367 (+ 238us) server_negotiation.cc:407] Connection header received
0620 12:14:33.192466 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:33.192469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:33.192525 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:33.192633 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:33.193186 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.193773 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.194721 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.194957 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.195036 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:33.198660 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:34807: BlockingRecv error: recv got EOF from 127.24.141.3:34807 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 12:14:33 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:33 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:33.751060 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:33.744658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:40251)
0620 12:14:33.744939 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:33.744954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:33.745051 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:33.745431 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:33.745434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:33.745646 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:33.745776 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:33.745782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.746669 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.746672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:33.747290 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:33.747300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.747791 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.747795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:33.747870 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:33.748453 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:33.748465 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:33.750909 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
W20260620 12:14:33.751291 1880 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:33.744809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:40251 (local address 127.24.141.62:43141)
0620 12:14:33.745140 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:33.745144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:33.745159 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:14:33.745203 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:33.745205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:33.745261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:33.745360 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:33.745914 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.746534 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.747423 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.747665 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.747752 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:33.751116 (+ 3364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:40251: BlockingRecv error: recv got EOF from 127.24.141.1:40251 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
Jun 20 12:14:33 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:33 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:33.990280 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:33.983645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:39725)
0620 12:14:33.983950 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:33.983965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:33.984094 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:33.984438 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:33.984442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:33.984630 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:33.984749 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:33.984754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.985794 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.985797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:33.986454 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:33.986467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.987017 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.987020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:33.987121 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:33.987771 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:33.987787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:33.990084 (+ 2297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
W20260620 12:14:33.990346 1880 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:33.983781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:39725 (local address 127.24.141.62:43141)
0620 12:14:33.983969 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:33.983973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:33.984042 (+ 69us) server_negotiation.cc:407] Connection header received
0620 12:14:33.984189 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:33.984191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:33.984246 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:33.984349 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:33.984868 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.985636 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.986583 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:33.986869 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:33.986976 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:33.990234 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:39725: BlockingRecv error: recv got EOF from 127.24.141.2:39725 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 12:14:34 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:34 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:34.205926 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:34.199551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:33471)
0620 12:14:34.199900 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:34.199920 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:34.200067 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:34.200355 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:34.200358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:34.200607 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:34.200732 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:34.200738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.201565 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.201568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:34.202202 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:34.202211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.202717 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.202720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:34.202794 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:34.203501 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:34.203514 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:34.205770 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
W20260620 12:14:34.206228 1880 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:34.199659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:33471 (local address 127.24.141.62:43141)
0620 12:14:34.199843 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:34.199848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:34.200017 (+ 169us) server_negotiation.cc:407] Connection header received
0620 12:14:34.200161 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:34.200164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:34.200217 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:34.200300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:34.200836 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.201439 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.202352 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.202586 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.202675 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:34.206061 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:33471: BlockingRecv error: recv got EOF from 127.24.141.3:33471 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 12:14:34 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:34 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:34.758159 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:34.752159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:39329)
0620 12:14:34.752555 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:34.752574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:34.752718 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:34.752983 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:34.752986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:34.753226 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:34.753321 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:34.753344 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.754111 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.754115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:34.754677 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:34.754685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.755133 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.755137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:34.755217 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:34.755818 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:34.755831 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:34.757964 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":115,"threads_started":1}
W20260620 12:14:34.758385 1884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:34.752240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39329 (local address 127.24.141.62:43141)
0620 12:14:34.752545 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:34.752550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:34.752650 (+ 100us) server_negotiation.cc:407] Connection header received
0620 12:14:34.752809 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:34.752812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:34.752861 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:34.752937 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:34.753457 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.753998 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.754793 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.755019 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.755107 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:34.758202 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:39329: BlockingRecv error: recv got EOF from 127.24.141.1:39329 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
Jun 20 12:14:34 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:34 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:34.998214 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:34.991602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:38263)
0620 12:14:34.991895 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:34.991910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:34.992015 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:34.992384 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:34.992387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:34.992582 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:34.992701 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:34.992707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.993718 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.993722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:34.994330 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:34.994342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.994863 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.994866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:34.994944 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:34.995593 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:34.995605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:34.998019 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
W20260620 12:14:34.998369 1884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:34.991784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:38263 (local address 127.24.141.62:43141)
0620 12:14:34.991995 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:34.991998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:34.992021 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:34.992127 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:34.992129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:34.992186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:34.992322 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:34.992882 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.993588 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.994485 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:34.994723 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:34.994806 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:34.998207 (+ 3401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:38263: BlockingRecv error: recv got EOF from 127.24.141.2:38263 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 12:14:35 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:35 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:35.214576 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:35.207098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:48549)
0620 12:14:35.207564 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:35.207585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:35.207764 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:35.208062 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:35.208065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:35.208309 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:35.208424 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:35.208430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.209386 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.209391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:35.210101 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:35.210114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.210567 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.210571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:35.210655 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:35.211289 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:35.211304 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:35.214370 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":130,"threads_started":1}
W20260620 12:14:35.214759 1884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:35.207319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:48549 (local address 127.24.141.62:43141)
0620 12:14:35.207564 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:35.207570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:35.207695 (+ 125us) server_negotiation.cc:407] Connection header received
0620 12:14:35.207878 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:35.207881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:35.207937 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:35.208019 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:35.208578 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.209271 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.210216 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.210450 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.210549 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:35.214633 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:48549: BlockingRecv error: recv got EOF from 127.24.141.3:48549 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Jun 20 12:14:35 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:35 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:35.765647 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:35.759184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:41807)
0620 12:14:35.759512 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:35.759530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:35.759639 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:35.759993 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:35.759996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:35.760208 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:35.760335 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:35.760340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.761167 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.761170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:35.761806 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:35.761815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.762370 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.762373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:35.762456 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:35.763082 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:35.763095 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:35.765478 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":102,"threads_started":1}
W20260620 12:14:35.765887 1888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:35.759280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:41807 (local address 127.24.141.62:43141)
0620 12:14:35.759571 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:35.759575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:35.759598 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:35.759779 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:35.759782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:35.759835 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:35.759950 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:35.760462 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.761016 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.761931 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:35.762182 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:35.762293 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:35.765723 (+ 3430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:41807: BlockingRecv error: recv got EOF from 127.24.141.1:41807 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":88,"threads_started":1}
Jun 20 12:14:36 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:36 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:36.006165 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:35.999237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:34243)
0620 12:14:35.999658 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:35.999676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:35.999843 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:36.000204 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:36.000207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:36.000405 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:36.000519 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:36.000525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.001599 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.001602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:36.002259 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:36.002272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.002859 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.002862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:36.002945 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:36.003623 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:36.003637 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:36.005962 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
W20260620 12:14:36.006397 1888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:35.999425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:34243 (local address 127.24.141.62:43141)
0620 12:14:35.999637 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:35.999641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:35.999868 (+ 227us) server_negotiation.cc:407] Connection header received
0620 12:14:35.999936 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:35.999939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:36.000000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:36.000092 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:36.000685 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.001426 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.002441 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.002696 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.002793 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:36.006165 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:34243: BlockingRecv error: recv got EOF from 127.24.141.2:34243 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 12:14:36 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:36 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:36.222591 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:36.215879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:39501)
0620 12:14:36.216460 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:36.216476 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:36.216680 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:36.217043 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:36.217047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:36.217350 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:36.217520 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:36.217528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.218341 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.218345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:36.218948 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:36.218957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.219402 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.219413 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:36.219499 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:36.220122 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:36.220135 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:36.222323 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":116,"threads_started":1}
W20260620 12:14:36.222637 1888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:36.216163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:39501 (local address 127.24.141.62:43141)
0620 12:14:36.216357 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:36.216364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:36.216589 (+ 225us) server_negotiation.cc:407] Connection header received
0620 12:14:36.216824 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:36.216828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:36.216893 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:36.216993 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:36.217647 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.218229 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.219069 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.219293 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.219380 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:36.222513 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:39501: BlockingRecv error: recv got EOF from 127.24.141.3:39501 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 12:14:36 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:36 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:36.773360 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:36.766736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:39243)
0620 12:14:36.767013 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:36.767029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:36.767136 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:36.767712 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:36.767715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:36.767954 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:36.768100 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:36.768108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.769165 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.769170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:36.769823 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:36.769838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.770418 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.770422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:36.770523 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:36.771187 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:36.771199 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:36.773188 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
W20260620 12:14:36.773566 1892 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:36.766909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39243 (local address 127.24.141.62:43141)
0620 12:14:36.767289 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:36.767294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:36.767315 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:14:36.767376 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:36.767379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:36.767444 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:36.767573 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:36.768268 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.768886 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.769989 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:36.770236 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:36.770331 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:36.773385 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:39243: BlockingRecv error: recv got EOF from 127.24.141.1:39243 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":151,"threads_started":1}
Jun 20 12:14:37 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:37 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:37.013619 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:37.007285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:60925)
0620 12:14:37.007648 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:37.007666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:37.007834 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:37.008113 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:37.008117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:37.008359 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:37.008480 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:37.008487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.009347 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.009351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:37.009930 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:37.009938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.010379 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.010383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:37.010474 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:37.011048 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:37.011061 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:37.013402 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
W20260620 12:14:37.013648 1892 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:37.007430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:60925 (local address 127.24.141.62:43141)
0620 12:14:37.007624 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:37.007629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:37.007770 (+ 141us) server_negotiation.cc:407] Connection header received
0620 12:14:37.007943 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:37.007946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:37.008001 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:37.008072 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:37.008600 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.009249 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.010040 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.010276 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.010361 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:37.013536 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:60925: BlockingRecv error: recv got EOF from 127.24.141.2:60925 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 12:14:37 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:37 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:37.230664 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:37.223887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:38883)
0620 12:14:37.224288 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:37.224307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:37.224459 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:37.224748 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:37.224751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:37.224978 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:37.225150 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:37.225159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.226171 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.226175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:37.226912 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:37.226925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.227428 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.227431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:37.227506 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:37.228088 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:37.228102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:37.230504 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
W20260620 12:14:37.230913 1892 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:37.224064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:38883 (local address 127.24.141.62:43141)
0620 12:14:37.224289 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:37.224293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:37.224403 (+ 110us) server_negotiation.cc:407] Connection header received
0620 12:14:37.224560 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:37.224563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:37.224614 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:37.224687 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:37.225308 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.226015 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.227043 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.227290 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.227401 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:37.230725 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:38883: BlockingRecv error: recv got EOF from 127.24.141.3:38883 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 12:14:37 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:37 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:37.780784 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:37.774425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:46239)
0620 12:14:37.774815 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:37.774837 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:37.774998 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:37.775357 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:37.775361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:37.775608 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:37.775730 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:37.775737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.776557 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.776560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:37.777183 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:37.777193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.777670 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.777674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:37.777757 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:37.778327 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:37.778339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:37.780606 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":99,"threads_started":1}
W20260620 12:14:37.780963 1896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:37.774524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:46239 (local address 127.24.141.62:43141)
0620 12:14:37.774811 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:37.774815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:37.774982 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:14:37.775119 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:37.775127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:37.775196 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:37.775293 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:37.775868 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.776431 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.777317 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:37.777549 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:37.777647 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:37.780834 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:46239: BlockingRecv error: recv got EOF from 127.24.141.1:46239 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":66,"threads_started":1}
Jun 20 12:14:38 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:38 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:38.026712 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:38.014797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:35089)
0620 12:14:38.015174 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:38.015193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:38.015344 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:38.015631 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:38.015634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:38.016049 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:38.016235 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:38.016247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.017295 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.017298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:38.017930 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:38.017941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.018439 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.018442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:38.018517 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:38.023766 (+ 5249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:38.023784 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:38.026573 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":114,"threads_started":1}
W20260620 12:14:38.026986 1896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:38.014966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:35089 (local address 127.24.141.62:43141)
0620 12:14:38.015178 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:38.015184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:38.015278 (+ 94us) server_negotiation.cc:407] Connection header received
0620 12:14:38.015456 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:38.015460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:38.015519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:38.015593 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:38.016445 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.017141 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.018066 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.018303 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.018390 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:38.026821 (+ 8431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:35089: BlockingRecv error: recv got EOF from 127.24.141.2:35089 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 12:14:38 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:38 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:38.238839 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:38.231752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:50199)
0620 12:14:38.232140 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:38.232157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:38.232312 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:38.232644 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:38.232648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:38.232889 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:38.233015 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:38.233021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.233977 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.233982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:38.234676 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:38.234687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.235181 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.235184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:38.235274 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:38.236119 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:38.236140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:38.238651 (+ 2511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":143,"threads_started":1}
W20260620 12:14:38.239096 1896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:38.231916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:50199 (local address 127.24.141.62:43141)
0620 12:14:38.232139 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:38.232144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:38.232247 (+ 103us) server_negotiation.cc:407] Connection header received
0620 12:14:38.232460 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:38.232463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:38.232520 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:38.232597 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:38.233179 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.233846 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.234823 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.235065 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.235156 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:38.238949 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:50199: BlockingRecv error: recv got EOF from 127.24.141.3:50199 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 12:14:38 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:38 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:38.788545 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:38.781941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:44861)
0620 12:14:38.782321 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:38.782339 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:38.782499 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:38.782845 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:38.782847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:38.783089 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:38.783219 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:38.783226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.784077 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.784080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:38.784779 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:38.784788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.785359 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.785362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:38.785470 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:38.786111 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:38.786124 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:38.788345 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
W20260620 12:14:38.788681 1900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:38.782155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:44861 (local address 127.24.141.62:43141)
0620 12:14:38.782423 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:38.782427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:38.782447 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:14:38.782628 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:38.782634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:38.782689 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:38.782791 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:38.783383 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.783953 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.784915 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:38.785208 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:38.785310 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:38.788542 (+ 3232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:44861: BlockingRecv error: recv got EOF from 127.24.141.1:44861 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
Jun 20 12:14:39 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:39 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:39.034556 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:39.027774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:51585)
0620 12:14:39.028210 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:39.028230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:39.028395 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:39.028680 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:39.028683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:39.028903 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:39.028997 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:39.029003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.029793 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.029796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:39.030361 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:39.030370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.030822 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.030826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:39.030905 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:39.031507 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:39.031519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:39.034382 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":99,"threads_started":1}
W20260620 12:14:39.034799 1900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:39.028052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:51585 (local address 127.24.141.62:43141)
0620 12:14:39.028257 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:39.028261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:39.028315 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:14:39.028506 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:39.028509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:39.028558 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:39.028636 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:39.029133 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.029679 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.030475 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.030696 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.030801 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:39.034627 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:51585: BlockingRecv error: recv got EOF from 127.24.141.2:51585 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 12:14:39 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:39 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:39.247349 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:39.240277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:42117)
0620 12:14:39.240618 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:39.240635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:39.240767 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:39.241292 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:39.241295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:39.241542 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:39.241677 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:39.241684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.242572 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.242575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:39.243381 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:39.243401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.244148 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.244152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:39.244245 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:39.244902 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:39.244919 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:39.247189 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
W20260620 12:14:39.247620 1900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:39.240500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:42117 (local address 127.24.141.62:43141)
0620 12:14:39.240658 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:39.240662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:39.240710 (+ 48us) server_negotiation.cc:407] Connection header received
0620 12:14:39.240924 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:39.240928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:39.240990 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:39.241168 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:39.241834 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.242398 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.243623 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.243963 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.244086 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:39.247450 (+ 3364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:42117: BlockingRecv error: recv got EOF from 127.24.141.3:42117 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 12:14:39 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:39 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:39.796754 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:39.789820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:39861)
0620 12:14:39.790153 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:39.790170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:39.790281 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:39.790765 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:39.790768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:39.791042 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:39.791178 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:39.791186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.792196 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.792199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:39.792870 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:39.792879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.793531 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.793534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:39.793621 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:39.794230 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:39.794243 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:39.796574 (+ 2331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
W20260620 12:14:39.797040 1904 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:39.789977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39861 (local address 127.24.141.62:43141)
0620 12:14:39.790460 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:39.790465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:39.790488 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:14:39.790530 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:39.790533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:39.790585 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:39.790714 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:39.791358 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.792084 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.793047 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:39.793369 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:39.793471 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:39.796861 (+ 3390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:39861: BlockingRecv error: recv got EOF from 127.24.141.1:39861 (error 108)
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":104,"threads_started":1}
Jun 20 12:14:40 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:40 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:40.043393 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:40.036084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:41259)
0620 12:14:40.036559 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:40.036590 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:40.036759 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:40.037292 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:40.037295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:40.037534 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:40.037664 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:40.037671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.038561 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.038564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:40.039275 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:40.039285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.039949 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.039952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:40.040039 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:40.040703 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:40.040716 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:40.043164 (+ 2448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
W20260620 12:14:40.043776 1904 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:40.036147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:41259 (local address 127.24.141.62:43141)
0620 12:14:40.036301 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:40.036304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:40.036735 (+ 431us) server_negotiation.cc:407] Connection header received
0620 12:14:40.036850 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:40.036852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:40.037008 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:40.037248 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:40.037847 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.038408 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.039488 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.039780 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.039901 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:40.043612 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:41259: BlockingRecv error: recv got EOF from 127.24.141.2:41259 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 12:14:40 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:40 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.3@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:40.255427 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:40.248368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.3:46011)
0620 12:14:40.248728 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:40.248747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:40.248903 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:40.249247 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:40.249251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:40.249490 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:40.249642 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:40.249649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.250547 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.250551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:40.251293 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:40.251420 (+ 127us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.252115 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.252118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:40.252209 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:40.252872 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:40.252886 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:40.255233 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
W20260620 12:14:40.255534 1904 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:40.248520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:46011 (local address 127.24.141.62:43141)
0620 12:14:40.248698 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:40.248703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:40.248849 (+ 146us) server_negotiation.cc:407] Connection header received
0620 12:14:40.248998 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:40.249001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:40.249089 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:40.249179 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:40.249833 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.250392 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.251647 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.251942 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.252061 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:40.255428 (+ 3367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.3:46011: BlockingRecv error: recv got EOF from 127.24.141.3:46011 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 12:14:40 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:40 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.1@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:40.804180 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:40.798047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.1:49163)
0620 12:14:40.798389 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:40.798407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:40.798526 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:40.798806 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:40.798810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:40.799055 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:40.799188 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:40.799195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.800033 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.800037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:40.800628 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:40.800637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.801106 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.801109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:40.801191 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:40.801953 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:40.801967 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:40.803978 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
W20260620 12:14:40.804286 1908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:40.798121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:49163 (local address 127.24.141.62:43141)
0620 12:14:40.798406 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:40.798411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:40.798486 (+ 75us) server_negotiation.cc:407] Connection header received
0620 12:14:40.798614 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:40.798617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:40.798670 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:40.798767 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:40.799297 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.799921 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.800742 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:40.800977 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:40.801095 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:40.804149 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.1:49163: BlockingRecv error: recv got EOF from 127.24.141.1:49163 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":73,"threads_started":1}
Jun 20 12:14:41 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:14:41 dist-test-slave-rh5v krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.141.2@KRBTEST.COM for oryx/127.24.141.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:14:41.051757 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:41.044693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43141 (local address 127.24.141.2:39737)
0620 12:14:41.045126 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:41.045145 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:41.045316 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:41.045635 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:41.045638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:41.045878 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:41.046031 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:41.046037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.046931 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.046936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:41.047617 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:41.047630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.048270 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.048275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:41.048428 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:41.049280 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:41.049296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:41.051532 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.141.62:43141: Server oryx/127.24.141.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":83,"threads_started":1}
W20260620 12:14:41.051888 1908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:41.044867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:39737 (local address 127.24.141.62:43141)
0620 12:14:41.045111 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:41.045117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:41.045308 (+ 191us) server_negotiation.cc:407] Connection header received
0620 12:14:41.045413 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:41.045415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:41.045476 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:41.045588 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:41.046196 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.046764 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.047774 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.048118 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.048238 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:41.051755 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.141.2:39737: BlockingRecv error: recv got EOF from 127.24.141.2:39737 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 12:14:41.124233 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 1431
I20260620 12:14:41.131476 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 1585
I20260620 12:14:41.137624 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 1717
I20260620 12:14:41.144044 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 1360
2026-06-20T12:14:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18326 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T12:14:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:41Z Disabled control of system clock
I20260620 12:14:41.179955 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45757
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:45757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 12:14:41.319849 1913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:41.320261 1913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:41.320302 1913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:41.324997 1913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:41.325155 1913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:41.325169 1913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:41.325187 1913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:41.325205 1913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:41.330941 1913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:45757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:41.332540 1913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:41.334474 1913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:41.346066 1918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:41.346217 1913 server_base.cc:1034] running on GCE node
W20260620 12:14:41.346067 1921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:41.346066 1919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:41.347301 1913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:41.348758 1913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:41.350030 1913 hybrid_clock.cc:648] HybridClock initialized: now 1781957681350029 us; error 80 us; skew 500 ppm
I20260620 12:14:41.352751 1913 webserver.cc:460] Webserver started at http://127.24.141.62:41917/ using document root <none> and password file <none>
I20260620 12:14:41.353636 1913 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:41.353737 1913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:41.353989 1913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:41.356717 1913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "7133b071277f425fad82cf8363c6ade6"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "a01c03f8afff8e508787e576478a9abe"
server_key_iv: "25c827844027a1ee4ec64831489e0e50"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.357788 1913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "7133b071277f425fad82cf8363c6ade6"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "a01c03f8afff8e508787e576478a9abe"
server_key_iv: "25c827844027a1ee4ec64831489e0e50"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.363653 1913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:14:41.367168 1927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:41.369231 1913 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:41.369437 1913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "7133b071277f425fad82cf8363c6ade6"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "a01c03f8afff8e508787e576478a9abe"
server_key_iv: "25c827844027a1ee4ec64831489e0e50"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.369709 1913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:41.389318 1913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:41.390199 1913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:41.390412 1913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:41.399708 1913 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:45757
I20260620 12:14:41.399762 1979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:45757 every 8 connection(s)
I20260620 12:14:41.400974 1913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:41.405340 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:41.408962 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 1913
I20260620 12:14:41.409152 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:41.409489 25140 external_mini_cluster.cc:1428] Setting key 8a3629d285d5a47aadadcf5c6da0b094
I20260620 12:14:41.413493 1980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6: Bootstrap starting.
I20260620 12:14:41.416914 1980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:41.418026 1980 log.cc:824] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:41.421738 1980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6: No bootstrap required, opened a new log
I20260620 12:14:41.421803 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:41.411424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.24.141.62:45757)
0620 12:14:41.412334 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:41.412358 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:41.412476 (+ 118us) server_negotiation.cc:407] Connection header received
0620 12:14:41.413410 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:41.413435 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:41.413994 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:41.414358 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:41.414945 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.416160 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.417536 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.418151 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.418542 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:41.419364 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:41.419372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:41.419399 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:41.419920 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:41.419997 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:41.420023 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:41.420160 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:41.420421 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 12:14:41.420833 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":156,"threads_started":1}
I20260620 12:14:41.426746 1980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7133b071277f425fad82cf8363c6ade6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45757 } }
I20260620 12:14:41.427145 1980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:41.427199 1980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133b071277f425fad82cf8363c6ade6, State: Initialized, Role: FOLLOWER
I20260620 12:14:41.427765 1980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7133b071277f425fad82cf8363c6ade6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45757 } }
I20260620 12:14:41.427933 1980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:41.428005 1980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:41.428117 1980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:41.429461 1980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7133b071277f425fad82cf8363c6ade6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45757 } }
I20260620 12:14:41.429936 1980 leader_election.cc:302] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7133b071277f425fad82cf8363c6ade6; no voters:
I20260620 12:14:41.430398 1980 leader_election.cc:288] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:41.430645 1985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:41.431203 1985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133b071277f425fad82cf8363c6ade6, State: Running, Role: LEADER
I20260620 12:14:41.431773 1985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7133b071277f425fad82cf8363c6ade6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45757 } }
I20260620 12:14:41.432214 1980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:41.434118 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7133b071277f425fad82cf8363c6ade6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7133b071277f425fad82cf8363c6ade6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45757 } } }
I20260620 12:14:41.434080 1987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7133b071277f425fad82cf8363c6ade6. Latest consensus state: current_term: 1 leader_uuid: "7133b071277f425fad82cf8363c6ade6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7133b071277f425fad82cf8363c6ade6" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45757 } } }
I20260620 12:14:41.434321 1987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:41.434326 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6 [sys.catalog]: This master's current role is: LEADER
W20260620 12:14:41.438325 2000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:14:41.438465 2000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:14:41.438534 1999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:41.439714 1999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:41.446702 1999 catalog_manager.cc:1347] Generated new cluster ID: 99e78694c8244b91af1fa0f15c28669d
I20260620 12:14:41.446820 1999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:41.463878 1999 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:41.465440 1999 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:41.481734 1999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7133b071277f425fad82cf8363c6ade6: Generated new TSK 0
I20260620 12:14:41.482815 1999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:14:41.494071 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45757
--builtin_ntp_servers=127.24.141.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 12:14:41.628755 2004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:41.629254 2004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:41.629320 2004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:41.634245 2004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:41.634346 2004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:41.634433 2004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:41.640084 2004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:41.641666 2004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:41.643641 2004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:41.655889 2009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:41.655864 2012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:41.656124 2010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:41.656396 2004 server_base.cc:1034] running on GCE node
I20260620 12:14:41.656987 2004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:41.657859 2004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:41.659099 2004 hybrid_clock.cc:648] HybridClock initialized: now 1781957681659066 us; error 53 us; skew 500 ppm
I20260620 12:14:41.661585 2004 webserver.cc:460] Webserver started at http://127.24.141.1:36513/ using document root <none> and password file <none>
I20260620 12:14:41.662312 2004 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:41.662398 2004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:41.662624 2004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:41.665103 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "686c859e0e614321bd4c199864bd446f"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "9128f44e30bbea33b3b067d0b9529c06"
server_key_iv: "630f370032b932c75ff24caf17a53632"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.665797 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "686c859e0e614321bd4c199864bd446f"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "9128f44e30bbea33b3b067d0b9529c06"
server_key_iv: "630f370032b932c75ff24caf17a53632"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.670619 2004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:41.674463 2018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:41.676512 2004 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:41.676716 2004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "686c859e0e614321bd4c199864bd446f"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "9128f44e30bbea33b3b067d0b9529c06"
server_key_iv: "630f370032b932c75ff24caf17a53632"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.676882 2004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:41.702091 2004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:41.702903 2004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:41.703110 2004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:41.703786 2004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:41.704934 2004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:41.704990 2004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:41.705049 2004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:41.705116 2004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 12:14:41.715478 2004 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:44305
I20260620 12:14:41.715499 2132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:44305 every 8 connection(s)
I20260620 12:14:41.716542 2004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:41.723345 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2004
I20260620 12:14:41.723501 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:41.723780 25140 external_mini_cluster.cc:1428] Setting key bb02de641a91c019999a4dfa9378b62c
I20260620 12:14:41.726230 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45757
--builtin_ntp_servers=127.24.141.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 12:14:41.729327 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:41.719053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:56673 (local address 127.24.141.62:45757)
0620 12:14:41.719273 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:41.719279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:41.720893 (+ 1614us) server_negotiation.cc:407] Connection header received
0620 12:14:41.722989 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:41.722993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:41.723091 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:41.723173 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:41.724335 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.725093 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.726669 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.727074 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.727195 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:41.728730 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:41.728734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:41.728736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:41.728750 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:41.728775 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:41.728783 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:41.728866 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:41.729105 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 12:14:41.729180 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 12:14:41.730461 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:41.719575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45757 (local address 127.24.141.1:56673)
0620 12:14:41.720656 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:41.720731 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:41.722709 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:41.723408 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:41.723417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:41.723558 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:41.724198 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:41.724218 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.725315 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.725321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:41.726511 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:41.726532 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.727259 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.727262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:41.727454 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:41.728310 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:41.728313 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:41.728389 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:41.728504 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:41.728899 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:41.728907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:41.728919 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 12:14:41.729112 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 12:14:41.729512 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":174,"threads_started":1}
I20260620 12:14:41.732149 2133 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45757
I20260620 12:14:41.732564 2133 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:41.733584 2133 heartbeater.cc:511] Master 127.24.141.62:45757 requested a full tablet report, sending...
I20260620 12:14:41.735819 1941 ts_manager.cc:194] Registered new tserver with Master: 686c859e0e614321bd4c199864bd446f (127.24.141.1:44305)
I20260620 12:14:41.737280 1941 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.1:56673
W20260620 12:14:41.856587 2136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:41.856942 2136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:41.856997 2136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:41.861316 2136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:41.861399 2136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:41.861490 2136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:41.866683 2136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:41.868202 2136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:41.870070 2136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:41.882210 2142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:41.882225 2141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:41.882565 2144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:41.882562 2136 server_base.cc:1034] running on GCE node
I20260620 12:14:41.883227 2136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:41.884011 2136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:41.885165 2136 hybrid_clock.cc:648] HybridClock initialized: now 1781957681885122 us; error 63 us; skew 500 ppm
I20260620 12:14:41.887629 2136 webserver.cc:460] Webserver started at http://127.24.141.2:36331/ using document root <none> and password file <none>
I20260620 12:14:41.888391 2136 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:41.888468 2136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:41.888706 2136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:41.891373 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "913676ed18e84b84a7520ef1450e1122"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "e9d81b76af316871d878bc4d0f2e27c3"
server_key_iv: "3275feb113f067bc189fcf184251d814"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.892061 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "913676ed18e84b84a7520ef1450e1122"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "e9d81b76af316871d878bc4d0f2e27c3"
server_key_iv: "3275feb113f067bc189fcf184251d814"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.897094 2136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 12:14:41.900924 2150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:41.902863 2136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:41.903049 2136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "913676ed18e84b84a7520ef1450e1122"
format_stamp: "Formatted at 2026-06-20 12:14:41 on dist-test-slave-rh5v"
server_key: "e9d81b76af316871d878bc4d0f2e27c3"
server_key_iv: "3275feb113f067bc189fcf184251d814"
server_key_version: "encryptionkey@0"
I20260620 12:14:41.903203 2136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:41.931913 2136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:41.932931 2136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:41.933182 2136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:41.934041 2136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:41.935388 2136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:41.935451 2136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:41.935513 2136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:41.935539 2136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:41.946856 2136 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:42629
I20260620 12:14:41.946898 2265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:42629 every 8 connection(s)
I20260620 12:14:41.948025 2136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:41.956337 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2136
I20260620 12:14:41.956459 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:41.956749 25140 external_mini_cluster.cc:1428] Setting key c3f2315c851b425bf252966725040de9
I20260620 12:14:41.959223 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45757
--builtin_ntp_servers=127.24.141.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 12:14:41.959678 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:41.950386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:55537 (local address 127.24.141.62:45757)
0620 12:14:41.950630 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:41.950635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:41.952118 (+ 1483us) server_negotiation.cc:407] Connection header received
0620 12:14:41.953758 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:41.953761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:41.953825 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:41.953927 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:41.954964 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.955578 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.957046 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.957352 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.957455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:41.959080 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:41.959083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:41.959085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:41.959098 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:41.959123 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:41.959131 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:41.959193 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:41.959422 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 12:14:41.959502 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 12:14:41.960907 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:41.950953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45757 (local address 127.24.141.2:55537)
0620 12:14:41.951918 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:41.951981 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:41.953505 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:41.954127 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:41.954135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:41.954237 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:41.954825 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:41.954839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.955759 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.955762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:41.956899 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:41.956914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:41.957748 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:41.957752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:41.957935 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:41.958707 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:41.958709 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:41.958777 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:41.958883 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:41.959229 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:41.959238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:41.959245 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:14:41.959404 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 12:14:41.959786 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":129,"threads_started":1}
I20260620 12:14:41.962781 2266 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45757
I20260620 12:14:41.963136 2266 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:41.963826 2266 heartbeater.cc:511] Master 127.24.141.62:45757 requested a full tablet report, sending...
I20260620 12:14:41.965538 1941 ts_manager.cc:194] Registered new tserver with Master: 913676ed18e84b84a7520ef1450e1122 (127.24.141.2:42629)
I20260620 12:14:41.966192 1941 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.2:55537
W20260620 12:14:42.091815 2269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:42.092242 2269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:42.092303 2269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:42.096892 2269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:42.096978 2269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:42.097093 2269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:42.102664 2269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:42.104250 2269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:42.106066 2269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:42.118145 2277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:42.118146 2275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:42.118146 2274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:42.118535 2269 server_base.cc:1034] running on GCE node
I20260620 12:14:42.119063 2269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:42.119941 2269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:42.121146 2269 hybrid_clock.cc:648] HybridClock initialized: now 1781957682121113 us; error 83 us; skew 500 ppm
I20260620 12:14:42.123620 2269 webserver.cc:460] Webserver started at http://127.24.141.3:42943/ using document root <none> and password file <none>
I20260620 12:14:42.124366 2269 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:42.124442 2269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:42.124696 2269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:42.127395 2269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "f0f96e48e9de4301b73236a54f09deff"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "72a9733003e9d145a131b138aa2cc436"
server_key_iv: "8328238953f5b0e286fe697e03670e36"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.128079 2269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "f0f96e48e9de4301b73236a54f09deff"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "72a9733003e9d145a131b138aa2cc436"
server_key_iv: "8328238953f5b0e286fe697e03670e36"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.133266 2269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 12:14:42.137455 2283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.139317 2269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:42.139497 2269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "f0f96e48e9de4301b73236a54f09deff"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "72a9733003e9d145a131b138aa2cc436"
server_key_iv: "8328238953f5b0e286fe697e03670e36"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.139653 2269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:42.165751 2269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:42.166689 2269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:42.166893 2269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:42.167668 2269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:42.168982 2269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:42.169036 2269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.169128 2269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:42.169155 2269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.180745 2269 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:43339
I20260620 12:14:42.180814 2396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:43339 every 8 connection(s)
I20260620 12:14:42.182008 2269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:42.188928 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2269
I20260620 12:14:42.189126 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:42.189512 25140 external_mini_cluster.cc:1428] Setting key 5883591a29c3fb6f8b1b9b128006ee1c
I20260620 12:14:42.195986 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:42.184744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:32795 (local address 127.24.141.62:45757)
0620 12:14:42.184967 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:42.184972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:42.186601 (+ 1629us) server_negotiation.cc:407] Connection header received
0620 12:14:42.188541 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:42.188545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:42.188624 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:42.188705 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:42.190144 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:42.190951 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:42.193119 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:42.193426 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:42.193540 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:42.195295 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:42.195299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:42.195302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:42.195320 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:42.195346 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:42.195355 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:42.195471 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:42.195729 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 12:14:42.195823 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 12:14:42.197316 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:42.185321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45757 (local address 127.24.141.3:32795)
0620 12:14:42.186371 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:42.186435 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:42.188185 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:42.188963 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:42.188973 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:42.189132 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:42.189870 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:42.189895 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:42.191160 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:42.191165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:42.192901 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:42.192921 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:42.193684 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:42.193689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:42.193951 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:42.194874 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:42.194876 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:42.194953 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:42.195074 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:42.195512 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:42.195522 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:42.195529 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:14:42.195696 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 12:14:42.196099 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":138,"threads_started":1}
I20260620 12:14:42.199059 2397 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45757
I20260620 12:14:42.199528 2397 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:42.200273 2397 heartbeater.cc:511] Master 127.24.141.62:45757 requested a full tablet report, sending...
I20260620 12:14:42.201836 1941 ts_manager.cc:194] Registered new tserver with Master: f0f96e48e9de4301b73236a54f09deff (127.24.141.3:43339)
I20260620 12:14:42.202471 1941 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.3:32795
I20260620 12:14:42.205767 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:14:42.209389 1983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:42.208237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.24.141.62:45757)
0620 12:14:42.208422 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:42.208427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:42.208596 (+ 169us) server_negotiation.cc:407] Connection header received
0620 12:14:42.208742 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:42.208745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:42.208799 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:42.208884 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:42.209221 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56288: BlockingRecv error: recv got EOF from 127.0.0.1:56288 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 12:14:42.209913 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2004
I20260620 12:14:42.215950 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2136
I20260620 12:14:42.222440 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2269
I20260620 12:14:42.228866 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 1913
2026-06-20T12:14:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1082 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T12:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:42Z Disabled control of system clock
I20260620 12:14:42.262243 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33963
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:33963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 12:14:42.397790 2408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:42.398185 2408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:42.398228 2408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:42.402773 2408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:42.402872 2408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:42.402887 2408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:42.402906 2408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:42.402922 2408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:42.408393 2408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:33963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:42.410125 2408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:42.411914 2408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:42.422631 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:42.422689 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:42.422739 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:42.422812 2408 server_base.cc:1034] running on GCE node
I20260620 12:14:42.423715 2408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:42.425276 2408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:42.426497 2408 hybrid_clock.cc:648] HybridClock initialized: now 1781957682426480 us; error 49 us; skew 500 ppm
I20260620 12:14:42.428905 2408 webserver.cc:460] Webserver started at http://127.24.141.62:40691/ using document root <none> and password file <none>
I20260620 12:14:42.429705 2408 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:42.429790 2408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:42.430040 2408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:42.432909 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "d2ed18b9266a452fadb14caa3fc0db33"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "dfeb5fc0b94f53e96d4bd061fcdd6419"
server_key_iv: "18da6e2ee0bcfd540f9edf59a757e217"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.433743 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "d2ed18b9266a452fadb14caa3fc0db33"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "dfeb5fc0b94f53e96d4bd061fcdd6419"
server_key_iv: "18da6e2ee0bcfd540f9edf59a757e217"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.438957 2408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:14:42.443142 2422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.445467 2408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 12:14:42.445696 2408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "d2ed18b9266a452fadb14caa3fc0db33"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "dfeb5fc0b94f53e96d4bd061fcdd6419"
server_key_iv: "18da6e2ee0bcfd540f9edf59a757e217"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.445900 2408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:42.452008 2408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:42.452878 2408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:42.453102 2408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:42.461879 2408 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:33963
I20260620 12:14:42.461905 2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:33963 every 8 connection(s)
I20260620 12:14:42.463172 2408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:42.467316 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:42.471455 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2408
I20260620 12:14:42.471596 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:42.471922 25140 external_mini_cluster.cc:1428] Setting key f5c175ea936579c34761fa4bd6f74e33
I20260620 12:14:42.475337 2475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33: Bootstrap starting.
I20260620 12:14:42.478283 2475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:42.479172 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:42.473728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.24.141.62:33963)
0620 12:14:42.474701 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:42.474722 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:42.474773 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:14:42.475662 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:42.475686 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:42.476184 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:42.476595 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:42.477505 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:42.477511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:42.477526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:42.477546 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:42.477621 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:42.477642 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:42.477763 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:42.477870 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 12:14:42.478187 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":191,"threads_started":1}
I20260620 12:14:42.479193 2475 log.cc:824] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:42.481961 2475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33: No bootstrap required, opened a new log
I20260620 12:14:42.485750 2475 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ed18b9266a452fadb14caa3fc0db33" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33963 } }
I20260620 12:14:42.486034 2475 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:42.486070 2475 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ed18b9266a452fadb14caa3fc0db33, State: Initialized, Role: FOLLOWER
I20260620 12:14:42.486562 2475 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ed18b9266a452fadb14caa3fc0db33" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33963 } }
I20260620 12:14:42.486707 2475 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:42.486792 2475 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:42.486905 2475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:42.487967 2475 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ed18b9266a452fadb14caa3fc0db33" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33963 } }
I20260620 12:14:42.488397 2475 leader_election.cc:302] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ed18b9266a452fadb14caa3fc0db33; no voters:
I20260620 12:14:42.488792 2475 leader_election.cc:288] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:42.489002 2480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:42.489377 2480 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [term 1 LEADER]: Becoming Leader. State: Replica: d2ed18b9266a452fadb14caa3fc0db33, State: Running, Role: LEADER
I20260620 12:14:42.489818 2480 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ed18b9266a452fadb14caa3fc0db33" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33963 } }
I20260620 12:14:42.490301 2475 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:42.492092 2481 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ed18b9266a452fadb14caa3fc0db33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ed18b9266a452fadb14caa3fc0db33" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33963 } } }
I20260620 12:14:42.492444 2481 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:42.492946 2494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:42.493142 2482 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ed18b9266a452fadb14caa3fc0db33. Latest consensus state: current_term: 1 leader_uuid: "d2ed18b9266a452fadb14caa3fc0db33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ed18b9266a452fadb14caa3fc0db33" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33963 } } }
I20260620 12:14:42.493620 2482 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33 [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:42.497264 2494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:42.504148 2494 catalog_manager.cc:1347] Generated new cluster ID: 2ee21fbaac974fec9e8fe50989ca23cd
I20260620 12:14:42.504278 2494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:42.526795 2494 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:42.527487 2494 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:42.549907 2494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2ed18b9266a452fadb14caa3fc0db33: Generated new TSK 0
I20260620 12:14:42.551224 2494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:14:42.565887 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33963
--builtin_ntp_servers=127.24.141.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 12:14:42.710790 2499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:42.711125 2499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:42.711185 2499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:42.715580 2499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:42.715668 2499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:42.715749 2499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:42.720927 2499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:42.722265 2499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:42.724164 2499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:42.735934 2504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:42.736033 2507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:42.736055 2499 server_base.cc:1034] running on GCE node
W20260620 12:14:42.735934 2505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:42.737011 2499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:42.737936 2499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:42.739132 2499 hybrid_clock.cc:648] HybridClock initialized: now 1781957682739114 us; error 44 us; skew 500 ppm
I20260620 12:14:42.741679 2499 webserver.cc:460] Webserver started at http://127.24.141.1:33175/ using document root <none> and password file <none>
I20260620 12:14:42.742456 2499 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:42.742533 2499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:42.742758 2499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:42.745543 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "eb9c4132f7e44c08a425dcdc573c0daa"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "03db8f67bf5bab617f87ed9736e936ae"
server_key_iv: "f0839265bcb6268e5a9d9d2dbfda8e4a"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.746210 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "eb9c4132f7e44c08a425dcdc573c0daa"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "03db8f67bf5bab617f87ed9736e936ae"
server_key_iv: "f0839265bcb6268e5a9d9d2dbfda8e4a"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.751509 2499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:14:42.755934 2513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.757851 2499 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:42.758024 2499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "eb9c4132f7e44c08a425dcdc573c0daa"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "03db8f67bf5bab617f87ed9736e936ae"
server_key_iv: "f0839265bcb6268e5a9d9d2dbfda8e4a"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.758194 2499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:42.764070 2499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:42.764966 2499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:42.765193 2499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:42.765933 2499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:42.767129 2499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:42.767180 2499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.767242 2499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:42.767270 2499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.778671 2499 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:37041
I20260620 12:14:42.778717 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:37041 every 8 connection(s)
I20260620 12:14:42.779889 2499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:42.785987 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2499
I20260620 12:14:42.786100 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:42.786402 25140 external_mini_cluster.cc:1428] Setting key 29f1a54d9571814b55adc7bd1cc31c84
I20260620 12:14:42.788030 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:42.782336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:43771 (local address 127.24.141.62:33963)
0620 12:14:42.782516 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:42.782521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:42.784051 (+ 1530us) server_negotiation.cc:407] Connection header received
0620 12:14:42.785601 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:42.785604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:42.785682 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:42.785766 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:42.787441 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:42.787446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:42.787448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:42.787465 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:42.787495 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:42.787505 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:42.787601 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:42.787783 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 12:14:42.787838 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:14:42.788749 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33963
--builtin_ntp_servers=127.24.141.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 12:14:42.790313 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:42.782904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33963 (local address 127.24.141.1:43771)
0620 12:14:42.783858 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:42.783915 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:42.785349 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:42.786149 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:42.786162 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:42.786298 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:42.787045 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:42.787046 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:42.787100 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:42.787212 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:42.787618 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:42.787627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:42.787633 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:14:42.787856 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 12:14:42.789202 (+ 1346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":158,"threads_started":1}
I20260620 12:14:42.791981 2627 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33963
I20260620 12:14:42.792336 2627 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:42.792973 2627 heartbeater.cc:511] Master 127.24.141.62:33963 requested a full tablet report, sending...
I20260620 12:14:42.795305 2439 ts_manager.cc:194] Registered new tserver with Master: eb9c4132f7e44c08a425dcdc573c0daa (127.24.141.1:37041)
W20260620 12:14:42.917999 2630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:42.918322 2630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:42.918383 2630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:42.922680 2630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:42.922761 2630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:42.922847 2630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:42.928040 2630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:42.929244 2630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:42.930691 2630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:42.941792 2635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:42.941855 2630 server_base.cc:1034] running on GCE node
W20260620 12:14:42.941800 2636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:42.941800 2638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:42.942937 2630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:42.943823 2630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:42.945019 2630 hybrid_clock.cc:648] HybridClock initialized: now 1781957682945007 us; error 49 us; skew 500 ppm
I20260620 12:14:42.947388 2630 webserver.cc:460] Webserver started at http://127.24.141.2:34205/ using document root <none> and password file <none>
I20260620 12:14:42.948130 2630 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:42.948204 2630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:42.948443 2630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:42.951224 2630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "5f62b69ca5ac4511a04110cd1b35d8df"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "8f7369a54abef1be89f1fc0a8dcc226e"
server_key_iv: "789a73bb9093e20600eabf1203f805a4"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.951906 2630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "5f62b69ca5ac4511a04110cd1b35d8df"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "8f7369a54abef1be89f1fc0a8dcc226e"
server_key_iv: "789a73bb9093e20600eabf1203f805a4"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.956745 2630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 12:14:42.960453 2644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.962448 2630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 12:14:42.962651 2630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "5f62b69ca5ac4511a04110cd1b35d8df"
format_stamp: "Formatted at 2026-06-20 12:14:42 on dist-test-slave-rh5v"
server_key: "8f7369a54abef1be89f1fc0a8dcc226e"
server_key_iv: "789a73bb9093e20600eabf1203f805a4"
server_key_version: "encryptionkey@0"
I20260620 12:14:42.962808 2630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:42.968828 2630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:42.969791 2630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:42.970001 2630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:42.970742 2630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:42.972163 2630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:42.972328 2630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.972425 2630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:42.972473 2630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:42.983760 2630 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:35767
I20260620 12:14:42.983913 2758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:35767 every 8 connection(s)
I20260620 12:14:42.985116 2630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:42.987142 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2630
I20260620 12:14:42.987274 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:42.987572 25140 external_mini_cluster.cc:1428] Setting key a559438f6094db94a3dbd620a7e60844
I20260620 12:14:42.990353 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33963
--builtin_ntp_servers=127.24.141.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 12:14:42.994477 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:42.987928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:38245 (local address 127.24.141.62:33963)
0620 12:14:42.988118 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:42.988123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:42.989961 (+ 1838us) server_negotiation.cc:407] Connection header received
0620 12:14:42.991837 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:42.991840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:42.991893 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:42.991972 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:42.993822 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:42.993828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:42.993831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:42.993856 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:42.993890 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:42.993902 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:42.994044 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:42.994267 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 12:14:42.994322 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:14:42.995901 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:42.988607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33963 (local address 127.24.141.2:38245)
0620 12:14:42.989770 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:42.989831 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:42.991522 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:42.992184 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:42.992193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:42.992315 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:42.993170 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:42.993172 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:42.993241 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:42.993392 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:42.994056 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:42.994068 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:42.994076 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:14:42.994343 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 12:14:42.994696 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":231,"threads_started":1}
I20260620 12:14:42.997757 2759 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33963
I20260620 12:14:42.998160 2759 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:42.998806 2759 heartbeater.cc:511] Master 127.24.141.62:33963 requested a full tablet report, sending...
I20260620 12:14:43.000473 2439 ts_manager.cc:194] Registered new tserver with Master: 5f62b69ca5ac4511a04110cd1b35d8df (127.24.141.2:35767)
W20260620 12:14:43.114375 2762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:43.114707 2762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:43.114755 2762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:43.119046 2762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:43.119149 2762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:43.119216 2762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:43.124279 2762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:43.125506 2762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:43.126981 2762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:43.138223 2767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:43.138223 2770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:43.138216 2768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:43.138307 2762 server_base.cc:1034] running on GCE node
I20260620 12:14:43.139043 2762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:43.139809 2762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:43.141036 2762 hybrid_clock.cc:648] HybridClock initialized: now 1781957683140999 us; error 46 us; skew 500 ppm
I20260620 12:14:43.148763 2762 webserver.cc:460] Webserver started at http://127.24.141.3:36563/ using document root <none> and password file <none>
I20260620 12:14:43.149545 2762 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:43.149629 2762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:43.149873 2762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:43.152436 2762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "c19c5fa2981d4d06b8c84267e286f791"
format_stamp: "Formatted at 2026-06-20 12:14:43 on dist-test-slave-rh5v"
server_key: "b2619c0f0e5ee5b6449c73151a28d0ff"
server_key_iv: "6811c820d6d9f8367b3c56108e95fc6a"
server_key_version: "encryptionkey@0"
I20260620 12:14:43.153654 2762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "c19c5fa2981d4d06b8c84267e286f791"
format_stamp: "Formatted at 2026-06-20 12:14:43 on dist-test-slave-rh5v"
server_key: "b2619c0f0e5ee5b6449c73151a28d0ff"
server_key_iv: "6811c820d6d9f8367b3c56108e95fc6a"
server_key_version: "encryptionkey@0"
I20260620 12:14:43.158526 2762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 12:14:43.161814 2776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:43.163554 2762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:14:43.163735 2762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "c19c5fa2981d4d06b8c84267e286f791"
format_stamp: "Formatted at 2026-06-20 12:14:43 on dist-test-slave-rh5v"
server_key: "b2619c0f0e5ee5b6449c73151a28d0ff"
server_key_iv: "6811c820d6d9f8367b3c56108e95fc6a"
server_key_version: "encryptionkey@0"
I20260620 12:14:43.163880 2762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:43.169025 2762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:43.169828 2762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:43.170017 2762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:43.170634 2762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:43.171725 2762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:43.171778 2762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:43.171844 2762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:43.171873 2762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:43.183372 2762 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:36771
I20260620 12:14:43.183393 2891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:36771 every 8 connection(s)
I20260620 12:14:43.184538 2762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:43.190070 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2762
I20260620 12:14:43.190230 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:43.190624 25140 external_mini_cluster.cc:1428] Setting key 984bb6252474cf9c6eb6593f3002fad5
I20260620 12:14:43.192787 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:43.187180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:58997 (local address 127.24.141.62:33963)
0620 12:14:43.187397 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:43.187401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:43.188865 (+ 1464us) server_negotiation.cc:407] Connection header received
0620 12:14:43.190519 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:43.190525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:43.190600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:43.190746 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:43.192256 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:43.192261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:43.192264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:14:43.192280 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:43.192309 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:14:43.192318 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:43.192395 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:43.192576 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 12:14:43.192629 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 12:14:43.193960 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:43.187671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33963 (local address 127.24.141.3:58997)
0620 12:14:43.188651 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:43.188715 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:43.190091 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:43.191044 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:43.191056 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:43.191213 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:43.191897 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:14:43.191898 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:43.191956 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:14:43.192055 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:43.192406 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:43.192414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:43.192421 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:14:43.192604 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 12:14:43.192873 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":171,"threads_started":1}
I20260620 12:14:43.195847 2892 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33963
I20260620 12:14:43.196203 2892 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:43.196718 2892 heartbeater.cc:511] Master 127.24.141.62:33963 requested a full tablet report, sending...
I20260620 12:14:43.198580 2439 ts_manager.cc:194] Registered new tserver with Master: c19c5fa2981d4d06b8c84267e286f791 (127.24.141.3:36771)
I20260620 12:14:43.206648 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:14:43.210734 2478 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:14:43.209354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.24.141.62:33963)
0620 12:14:43.209568 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:43.209573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:43.209700 (+ 127us) server_negotiation.cc:407] Connection header received
0620 12:14:43.210100 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:43.210104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:43.210181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:43.210262 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:43.210589 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34352: BlockingRecv error: recv got EOF from 127.0.0.1:34352 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 12:14:43.211414 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2499
I20260620 12:14:43.217702 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2630
I20260620 12:14:43.223381 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2762
I20260620 12:14:43.228998 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2408
2026-06-20T12:14:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (999 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:43 dist-test-slave-rh5v krb5kdc[2902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:43 dist-test-slave-rh5v krb5kdc[2902](info): set up 2 sockets
Jun 20 12:14:43 dist-test-slave-rh5v krb5kdc[2902](info): commencing operation
krb5kdc: starting...
W20260620 12:14:44.202471 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:44 dist-test-slave-rh5v krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:14:44.340289 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:39397
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:39397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:44.476564 2915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:44.476948 2915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:44.477011 2915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:44.481501 2915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:14:44.481604 2915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:44.481632 2915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:44.481654 2915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:14:44.481675 2915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:14:44.487800 2915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:39397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:39397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:44.489358 2915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:44.490777 2915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:44.502015 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:44.501977 2921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:44.501987 2920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:44.502202 2915 server_base.cc:1034] running on GCE node
I20260620 12:14:44.503024 2915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:44.504434 2915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:44.505654 2915 hybrid_clock.cc:648] HybridClock initialized: now 1781957684505620 us; error 52 us; skew 500 ppm
Jun 20 12:14:44 dist-test-slave-rh5v krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957684, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:44.509232 2915 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:14:44.510607 2915 webserver.cc:460] Webserver started at http://127.24.141.62:33151/ using document root <none> and password file <none>
I20260620 12:14:44.511365 2915 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:44.511448 2915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:44.511729 2915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:44.514756 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "dad43fd98dd54d779d5037a6d498bc4e"
format_stamp: "Formatted at 2026-06-20 12:14:44 on dist-test-slave-rh5v"
server_key: "d1f7fc23b58eee7bf73e06285efb25bf"
server_key_iv: "9a2c795daf24668fd3bdd30b5e87e138"
server_key_version: "encryptionkey@0"
I20260620 12:14:44.515506 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "dad43fd98dd54d779d5037a6d498bc4e"
format_stamp: "Formatted at 2026-06-20 12:14:44 on dist-test-slave-rh5v"
server_key: "d1f7fc23b58eee7bf73e06285efb25bf"
server_key_iv: "9a2c795daf24668fd3bdd30b5e87e138"
server_key_version: "encryptionkey@0"
I20260620 12:14:44.520845 2915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 12:14:44.524942 2930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:44.527029 2915 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:14:44.527262 2915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "dad43fd98dd54d779d5037a6d498bc4e"
format_stamp: "Formatted at 2026-06-20 12:14:44 on dist-test-slave-rh5v"
server_key: "d1f7fc23b58eee7bf73e06285efb25bf"
server_key_iv: "9a2c795daf24668fd3bdd30b5e87e138"
server_key_version: "encryptionkey@0"
I20260620 12:14:44.527413 2915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:44.544644 2915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:44.549257 2915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:44.549710 2915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:44.558266 2915 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:39397
I20260620 12:14:44.558312 2982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:39397 every 8 connection(s)
I20260620 12:14:44.559470 2915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:14:44.563678 2983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:44.569727 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 2915
I20260620 12:14:44.569854 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:14:44.570142 25140 external_mini_cluster.cc:1428] Setting key fbddd6099fa4c451dd142c0274d10f95
I20260620 12:14:44.571501 2983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e: Bootstrap starting.
I20260620 12:14:44.574602 2983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:44.575587 2983 log.cc:824] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:44.578152 2983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e: No bootstrap required, opened a new log
Jun 20 12:14:44 dist-test-slave-rh5v krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:44.583050 2983 raft_consensus.cc:348] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad43fd98dd54d779d5037a6d498bc4e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 39397 } }
I20260620 12:14:44.583487 2983 raft_consensus.cc:374] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:44.583542 2983 raft_consensus.cc:729] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad43fd98dd54d779d5037a6d498bc4e, State: Initialized, Role: FOLLOWER
I20260620 12:14:44.584165 2983 consensus_queue.cc:260] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad43fd98dd54d779d5037a6d498bc4e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 39397 } }
I20260620 12:14:44.584343 2983 raft_consensus.cc:388] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:14:44.584430 2983 raft_consensus.cc:482] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:14:44.584538 2983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:44.585973 2983 raft_consensus.cc:504] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad43fd98dd54d779d5037a6d498bc4e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 39397 } }
I20260620 12:14:44.586422 2983 leader_election.cc:302] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dad43fd98dd54d779d5037a6d498bc4e; no voters:
I20260620 12:14:44.586853 2983 leader_election.cc:288] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:14:44.587059 2988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:44.587368 2988 raft_consensus.cc:686] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [term 1 LEADER]: Becoming Leader. State: Replica: dad43fd98dd54d779d5037a6d498bc4e, State: Running, Role: LEADER
I20260620 12:14:44.587833 2988 consensus_queue.cc:237] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad43fd98dd54d779d5037a6d498bc4e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 39397 } }
I20260620 12:14:44.588172 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:44.571779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.24.141.62:39397)
0620 12:14:44.572473 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:44.572492 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:44.572548 (+ 56us) server_negotiation.cc:407] Connection header received
0620 12:14:44.573504 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:44.573530 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:44.573993 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:44.574713 (+ 720us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:44.575577 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:44.576839 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:44.577886 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:44.578534 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:44.578725 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:44.581770 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:44.581779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:44.581806 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:44.581856 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:44.584115 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:44.584800 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:44.584808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:44.584815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:44.584992 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:44.585325 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:44.585328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:44.585330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:44.585972 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:44.586204 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:44.586516 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 12:14:44.586905 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":195,"threads_started":1}
I20260620 12:14:44.588536 2983 sys_catalog.cc:564] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:14:44.590215 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dad43fd98dd54d779d5037a6d498bc4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad43fd98dd54d779d5037a6d498bc4e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 39397 } } }
I20260620 12:14:44.590404 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [sys.catalog]: This master's current role is: LEADER
I20260620 12:14:44.590816 2990 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dad43fd98dd54d779d5037a6d498bc4e. Latest consensus state: current_term: 1 leader_uuid: "dad43fd98dd54d779d5037a6d498bc4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad43fd98dd54d779d5037a6d498bc4e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 39397 } } }
I20260620 12:14:44.590916 2990 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e [sys.catalog]: This master's current role is: LEADER
W20260620 12:14:44.595420 3003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:14:44.595534 3003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:14:44.595606 2997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:14:44.596410 2997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:14:44.603168 2997 catalog_manager.cc:1347] Generated new cluster ID: b6bd53dcb96040468eeba4ab04eb87df
I20260620 12:14:44.603298 2997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:14:44.618914 2997 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:14:44.620698 2997 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:14:44.631510 2997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e: Generated new TSK 0
I20260620 12:14:44.632560 2997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:14:44.665568 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:39397
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:44.798669 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:44.799003 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:44.799052 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:44.803409 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:44.803491 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:44.803555 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:14:44.809296 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:44.810478 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:44.811554 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:44.823694 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:44.823705 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:44.823640 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:44.824055 3009 server_base.cc:1034] running on GCE node
I20260620 12:14:44.824620 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:44.825452 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:44.826684 3009 hybrid_clock.cc:648] HybridClock initialized: now 1781957684826640 us; error 65 us; skew 500 ppm
Jun 20 12:14:44 dist-test-slave-rh5v krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957684, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:44.830389 3009 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:14:44.831661 3009 webserver.cc:460] Webserver started at http://127.24.141.1:37677/ using document root <none> and password file <none>
I20260620 12:14:44.832335 3009 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:44.832417 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:44.832664 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:44.835083 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "8c20809db725438db1323f78abf9cb93"
format_stamp: "Formatted at 2026-06-20 12:14:44 on dist-test-slave-rh5v"
server_key: "2e856150a61d33f600d90ae1215b592e"
server_key_iv: "18b549de8a2add8fb99d7b0fc8251908"
server_key_version: "encryptionkey@0"
I20260620 12:14:44.835708 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "8c20809db725438db1323f78abf9cb93"
format_stamp: "Formatted at 2026-06-20 12:14:44 on dist-test-slave-rh5v"
server_key: "2e856150a61d33f600d90ae1215b592e"
server_key_iv: "18b549de8a2add8fb99d7b0fc8251908"
server_key_version: "encryptionkey@0"
I20260620 12:14:44.840518 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 12:14:44.844099 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:44.845945 3009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 12:14:44.846149 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "8c20809db725438db1323f78abf9cb93"
format_stamp: "Formatted at 2026-06-20 12:14:44 on dist-test-slave-rh5v"
server_key: "2e856150a61d33f600d90ae1215b592e"
server_key_iv: "18b549de8a2add8fb99d7b0fc8251908"
server_key_version: "encryptionkey@0"
I20260620 12:14:44.846295 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:44.862640 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:44.866271 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:44.866483 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:44.867148 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:44.868136 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:44.868188 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:44.868248 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:44.868278 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:44.878752 3009 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:40313
I20260620 12:14:44.878798 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:40313 every 8 connection(s)
I20260620 12:14:44.879840 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:14:44.883903 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 3009
I20260620 12:14:44.884066 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:14:44.884459 25140 external_mini_cluster.cc:1428] Setting key 04af4b7a8c3719dc2af320cb0b717304
Jun 20 12:14:44 dist-test-slave-rh5v krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957684, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:14:44.898535 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:44.882509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:52507 (local address 127.24.141.62:39397)
0620 12:14:44.882699 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:44.882704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:44.883892 (+ 1188us) server_negotiation.cc:407] Connection header received
0620 12:14:44.885852 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:44.885856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:44.885907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:44.886018 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:44.888298 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:44.889041 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:44.890679 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:44.891009 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:44.891113 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:44.894236 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:44.894240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:44.894242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:44.894272 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:44.895666 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:44.896455 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:44.896458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:44.896460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:44.896614 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:44.897023 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:44.897026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:44.897027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:44.897289 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:44.897404 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:44.898146 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 12:14:44.898330 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:14:44.899751 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:44.882921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:39397 (local address 127.24.141.1:52507)
0620 12:14:44.883607 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:44.883691 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:44.885588 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:44.886495 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:44.886506 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:44.887125 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:44.888026 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:44.888052 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:44.889292 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:44.889297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:44.890492 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:44.890514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:44.891180 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:44.891185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:44.891397 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:44.892037 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:44.892056 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:44.893989 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:44.895877 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:44.895888 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:44.895920 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:44.896306 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:44.896739 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:44.896742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:44.896744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:44.896928 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:44.897440 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:44.897447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:44.897821 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 12:14:44.898054 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 12:14:44.898518 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":160,"threads_started":1}
I20260620 12:14:44.901814 3138 heartbeater.cc:347] Connected to a master server at 127.24.141.62:39397
I20260620 12:14:44.902182 3138 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:44.902947 3138 heartbeater.cc:511] Master 127.24.141.62:39397 requested a full tablet report, sending...
I20260620 12:14:44.905158 2942 ts_manager.cc:194] Registered new tserver with Master: 8c20809db725438db1323f78abf9cb93 (127.24.141.1:40313)
I20260620 12:14:44.906543 2942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:52507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:14:44.917618 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:39397
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:45.050417 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:45.050777 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:45.050848 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:45.055828 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:45.056073 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:45.056250 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:14:45.062726 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:45.063974 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:45.065150 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:45.076742 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:45.076742 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:45.076792 3143 server_base.cc:1034] running on GCE node
W20260620 12:14:45.077095 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:45.077653 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:45.078599 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:45.079808 3143 hybrid_clock.cc:648] HybridClock initialized: now 1781957685079781 us; error 50 us; skew 500 ppm
Jun 20 12:14:45 dist-test-slave-rh5v krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:45.083595 3143 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:14:45.084960 3143 webserver.cc:460] Webserver started at http://127.24.141.2:39299/ using document root <none> and password file <none>
I20260620 12:14:45.085770 3143 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:45.085850 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:45.086073 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:45.088761 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "352500838ccb43ed81b904d0b282c3e6"
format_stamp: "Formatted at 2026-06-20 12:14:45 on dist-test-slave-rh5v"
server_key: "31cd119056b9e7c615c3050e59c6a2f8"
server_key_iv: "00cf75910956d4c4a0d32d3b04ca2d60"
server_key_version: "encryptionkey@0"
I20260620 12:14:45.089637 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "352500838ccb43ed81b904d0b282c3e6"
format_stamp: "Formatted at 2026-06-20 12:14:45 on dist-test-slave-rh5v"
server_key: "31cd119056b9e7c615c3050e59c6a2f8"
server_key_iv: "00cf75910956d4c4a0d32d3b04ca2d60"
server_key_version: "encryptionkey@0"
I20260620 12:14:45.094782 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 12:14:45.098608 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:45.100663 3143 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:14:45.100874 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "352500838ccb43ed81b904d0b282c3e6"
format_stamp: "Formatted at 2026-06-20 12:14:45 on dist-test-slave-rh5v"
server_key: "31cd119056b9e7c615c3050e59c6a2f8"
server_key_iv: "00cf75910956d4c4a0d32d3b04ca2d60"
server_key_version: "encryptionkey@0"
I20260620 12:14:45.101034 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:45.121923 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:45.125976 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:45.126192 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:45.127004 3143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:45.128145 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:45.128202 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:45.128252 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:45.128285 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:45.140157 3143 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:35169
I20260620 12:14:45.140182 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:35169 every 8 connection(s)
I20260620 12:14:45.141469 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:14:45.147382 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 3143
I20260620 12:14:45.147516 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:14:45.147797 25140 external_mini_cluster.cc:1428] Setting key 1be73bba7c93cdec3fe92f2473ec88d2
Jun 20 12:14:45 dist-test-slave-rh5v krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:14:45.160535 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.144004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:56893 (local address 127.24.141.62:39397)
0620 12:14:45.144213 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.144218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.145728 (+ 1510us) server_negotiation.cc:407] Connection header received
0620 12:14:45.147452 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.147457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.147531 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.147593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:45.149672 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.150444 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.152194 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.152569 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.152735 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.156206 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:45.156211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:45.156214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:45.156259 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:45.157827 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:45.158593 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:45.158596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:45.158597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:45.158704 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:45.159138 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:45.159140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:45.159142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:45.159359 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:45.159440 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.160234 (+ 794us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.160378 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 12:14:45.161731 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.144765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:39397 (local address 127.24.141.2:56893)
0620 12:14:45.145490 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.145551 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.147129 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.148054 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.148063 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.148619 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:45.149447 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.149468 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.150643 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.150647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.152014 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.152027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.152784 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.152788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.152988 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.153751 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:45.153775 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:45.155913 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:45.157989 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:45.157997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:45.158020 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:45.158427 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:45.158810 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:45.158814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:45.158815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:45.159034 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:45.159474 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:45.159483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:45.159914 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.160168 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.160528 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":233,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:14:45.163533 3272 heartbeater.cc:347] Connected to a master server at 127.24.141.62:39397
I20260620 12:14:45.163908 3272 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:45.164564 3272 heartbeater.cc:511] Master 127.24.141.62:39397 requested a full tablet report, sending...
I20260620 12:14:45.166190 2942 ts_manager.cc:194] Registered new tserver with Master: 352500838ccb43ed81b904d0b282c3e6 (127.24.141.2:35169)
I20260620 12:14:45.167073 2942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:56893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:14:45.184576 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:39397
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:14:45.319751 3277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:14:45.320123 3277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:14:45.320189 3277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:14:45.325322 3277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:14:45.325428 3277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:14:45.325517 3277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:14:45.332984 3277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:14:45.334476 3277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:14:45.335704 3277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:14:45.348241 3285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:45.348241 3282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:14:45.348318 3283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:14:45.348322 3277 server_base.cc:1034] running on GCE node
I20260620 12:14:45.349467 3277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:14:45.350528 3277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:14:45.351857 3277 hybrid_clock.cc:648] HybridClock initialized: now 1781957685351795 us; error 82 us; skew 500 ppm
Jun 20 12:14:45 dist-test-slave-rh5v krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:14:45.355549 3277 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:14:45.356802 3277 webserver.cc:460] Webserver started at http://127.24.141.3:46145/ using document root <none> and password file <none>
I20260620 12:14:45.357566 3277 fs_manager.cc:359] Metadata directory not provided
I20260620 12:14:45.357653 3277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:14:45.357889 3277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:14:45.360345 3277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "818ae906e8b3443d8aee6a383ae9f0dc"
format_stamp: "Formatted at 2026-06-20 12:14:45 on dist-test-slave-rh5v"
server_key: "7988865027770b472f75229d31c0484b"
server_key_iv: "d7cd54637595cc31d53a859f670d4b08"
server_key_version: "encryptionkey@0"
I20260620 12:14:45.361022 3277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "818ae906e8b3443d8aee6a383ae9f0dc"
format_stamp: "Formatted at 2026-06-20 12:14:45 on dist-test-slave-rh5v"
server_key: "7988865027770b472f75229d31c0484b"
server_key_iv: "d7cd54637595cc31d53a859f670d4b08"
server_key_version: "encryptionkey@0"
I20260620 12:14:45.366010 3277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:14:45.369539 3292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:45.371781 3277 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:14:45.372020 3277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "818ae906e8b3443d8aee6a383ae9f0dc"
format_stamp: "Formatted at 2026-06-20 12:14:45 on dist-test-slave-rh5v"
server_key: "7988865027770b472f75229d31c0484b"
server_key_iv: "d7cd54637595cc31d53a859f670d4b08"
server_key_version: "encryptionkey@0"
I20260620 12:14:45.372165 3277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:14:45.404748 3277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:14:45.409237 3277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:14:45.409483 3277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:14:45.410292 3277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:14:45.411413 3277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:14:45.411469 3277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:45.411540 3277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:14:45.411569 3277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:14:45.423416 3277 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:34781
I20260620 12:14:45.423460 3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:34781 every 8 connection(s)
I20260620 12:14:45.424698 3277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:14:45.434602 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 3277
I20260620 12:14:45.434769 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:14:45.435098 25140 external_mini_cluster.cc:1428] Setting key 53a2ac7a0d5d216d055f08b71bea6261
Jun 20 12:14:45 dist-test-slave-rh5v krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:14:45.443676 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.427212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:60041 (local address 127.24.141.62:39397)
0620 12:14:45.427471 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.427476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.428706 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 12:14:45.430391 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.430396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.430474 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.430631 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:45.432778 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.433440 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.435365 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.435676 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.435788 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.439588 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:45.439592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:45.439596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:45.439630 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:45.441120 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:45.441804 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:45.441809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:45.441811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:45.441893 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:45.442268 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:45.442271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:45.442273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:45.442521 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:45.442617 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.443305 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.443477 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 12:14:45.444847 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.427721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:39397 (local address 127.24.141.3:60041)
0620 12:14:45.428474 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.428543 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.430072 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.430923 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.430933 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.431707 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:14:45.432514 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.432540 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.433638 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.433642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.435097 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.435126 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.435988 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.436008 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.436303 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.437202 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:14:45.437226 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:14:45.439324 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:14:45.441277 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:45.441288 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:45.441309 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:45.441698 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:45.441991 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:14:45.441994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:14:45.441996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:14:45.442175 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:14:45.442632 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:14:45.442642 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:14:45.443027 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.443257 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.443725 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":164,"threads_started":1}
I20260620 12:14:45.446542 3406 heartbeater.cc:347] Connected to a master server at 127.24.141.62:39397
I20260620 12:14:45.446921 3406 heartbeater.cc:464] Registering TS with master...
I20260620 12:14:45.447568 3406 heartbeater.cc:511] Master 127.24.141.62:39397 requested a full tablet report, sending...
I20260620 12:14:45.449239 2942 ts_manager.cc:194] Registered new tserver with Master: 818ae906e8b3443d8aee6a383ae9f0dc (127.24.141.3:34781)
I20260620 12:14:45.450115 2942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:60041
I20260620 12:14:45.450923 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:14:45.461654 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.453459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.24.141.62:39397)
0620 12:14:45.453596 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.453602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.453807 (+ 205us) server_negotiation.cc:407] Connection header received
0620 12:14:45.453958 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.453962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.454015 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.454084 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:14:45.454873 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.455439 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.456311 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.456552 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.456647 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.458245 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:14:45.458250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:14:45.458253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:14:45.458284 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:14:45.459756 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:45.460221 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:45.460225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:45.460227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:45.460313 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:14:45.460575 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:14:45.460578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:14:45.460580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:14:45.460868 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:14:45.461005 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.461305 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.461449 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:14:45.466813 2942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:14:45.471100 2942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:14:45.491578 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.484236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.24.141.3:34781)
0620 12:14:45.485402 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.485414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.485436 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:14:45.485501 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.485506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.485655 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.485838 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:45.486168 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.487456 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.489568 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.490494 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.490683 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.490863 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.491240 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.491368 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":133,"threads_started":1}
I20260620 12:14:45.491778 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.483724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34781 (local address 127.0.0.1:36506)
0620 12:14:45.484810 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.484891 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.485107 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.485818 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.485825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.485849 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:45.486030 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.486038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.487829 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.487832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.489299 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.489313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.490674 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.490679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.490874 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.491040 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.491194 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.491401 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":61,"spinlock_wait_cycles":9728,"thread_start_us":92,"threads_started":1}
I20260620 12:14:45.492218 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.484622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.24.141.1:40313)
0620 12:14:45.485407 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.485415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.485652 (+ 237us) server_negotiation.cc:407] Connection header received
0620 12:14:45.485782 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.485788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.486038 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.486194 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:45.486387 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.487200 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.489207 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.490277 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.490446 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.490674 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.491874 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.492050 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":121,"threads_started":1}
I20260620 12:14:45.492309 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.483594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.24.141.2:35169)
0620 12:14:45.484140 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.484154 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.485287 (+ 1133us) server_negotiation.cc:407] Connection header received
0620 12:14:45.485387 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.485394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.485656 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.485838 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:45.486352 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.487496 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.489485 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.490431 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.490565 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.490690 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.491962 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.492125 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":218,"threads_started":1}
I20260620 12:14:45.493011 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.483382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:35169 (local address 127.0.0.1:51060)
0620 12:14:45.484811 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.484893 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.485117 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.486053 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.486056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.486077 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:45.486212 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.486222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.487643 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.487647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.489312 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.489319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.490755 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.490759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.490981 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.491048 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.491178 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.491385 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":298,"spinlock_wait_cycles":14976,"thread_start_us":141,"threads_started":1}
I20260620 12:14:45.493306 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.484414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:40313 (local address 127.0.0.1:48842)
0620 12:14:45.485550 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.485561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.485658 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.486187 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.486190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.486206 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:45.486299 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.486305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.487326 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.487330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.489049 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.489089 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.490755 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.490759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.490932 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.491630 (+ 698us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.491696 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.491786 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":961,"thread_start_us":121,"threads_started":1}
I20260620 12:14:45.495544 3340 tablet_service.cc:1467] Processing CreateTablet for tablet 7d22be1b7f1c4d1697c5449cb041c2a0 (DEFAULT_TABLE table=test-table [id=592c3d51e77243e28df746d3750bec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:45.496989 3340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d22be1b7f1c4d1697c5449cb041c2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:45.496717 3206 tablet_service.cc:1467] Processing CreateTablet for tablet 7d22be1b7f1c4d1697c5449cb041c2a0 (DEFAULT_TABLE table=test-table [id=592c3d51e77243e28df746d3750bec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:45.496717 3072 tablet_service.cc:1467] Processing CreateTablet for tablet 7d22be1b7f1c4d1697c5449cb041c2a0 (DEFAULT_TABLE table=test-table [id=592c3d51e77243e28df746d3750bec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:14:45.498183 3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d22be1b7f1c4d1697c5449cb041c2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:45.498178 3072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d22be1b7f1c4d1697c5449cb041c2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:14:45.504735 3424 tablet_bootstrap.cc:492] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Bootstrap starting.
I20260620 12:14:45.507720 3424 tablet_bootstrap.cc:654] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:45.509037 3425 tablet_bootstrap.cc:492] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: Bootstrap starting.
I20260620 12:14:45.509320 3424 log.cc:824] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:45.509431 3426 tablet_bootstrap.cc:492] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: Bootstrap starting.
I20260620 12:14:45.512167 3424 tablet_bootstrap.cc:492] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: No bootstrap required, opened a new log
I20260620 12:14:45.512322 3425 tablet_bootstrap.cc:654] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:45.512566 3424 ts_tablet_manager.cc:1397] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:14:45.512789 3426 tablet_bootstrap.cc:654] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: Neither blocks nor log segments found. Creating new log.
I20260620 12:14:45.513603 3425 log.cc:824] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:45.513697 3426 log.cc:824] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:14:45.516340 3426 tablet_bootstrap.cc:492] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: No bootstrap required, opened a new log
I20260620 12:14:45.516340 3425 tablet_bootstrap.cc:492] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: No bootstrap required, opened a new log
I20260620 12:14:45.516613 3425 ts_tablet_manager.cc:1397] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 12:14:45.516613 3426 ts_tablet_manager.cc:1397] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260620 12:14:45.516754 3424 raft_consensus.cc:348] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.517052 3424 raft_consensus.cc:374] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:45.517167 3424 raft_consensus.cc:729] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818ae906e8b3443d8aee6a383ae9f0dc, State: Initialized, Role: FOLLOWER
I20260620 12:14:45.517715 3424 consensus_queue.cc:260] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.518486 3406 heartbeater.cc:503] Master 127.24.141.62:39397 was elected leader, sending a full tablet report...
I20260620 12:14:45.518951 3424 ts_tablet_manager.cc:1428] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 12:14:45.521445 3426 raft_consensus.cc:348] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.521536 3425 raft_consensus.cc:348] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.521862 3425 raft_consensus.cc:374] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:45.521867 3426 raft_consensus.cc:374] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:14:45.521916 3425 raft_consensus.cc:729] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c20809db725438db1323f78abf9cb93, State: Initialized, Role: FOLLOWER
I20260620 12:14:45.521916 3426 raft_consensus.cc:729] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352500838ccb43ed81b904d0b282c3e6, State: Initialized, Role: FOLLOWER
I20260620 12:14:45.522557 3426 consensus_queue.cc:260] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.522557 3425 consensus_queue.cc:260] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.523572 3272 heartbeater.cc:503] Master 127.24.141.62:39397 was elected leader, sending a full tablet report...
I20260620 12:14:45.524065 3425 ts_tablet_manager.cc:1428] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: Time spent starting tablet: real 0.007s user 0.008s sys 0.001s
I20260620 12:14:45.524464 3138 heartbeater.cc:503] Master 127.24.141.62:39397 was elected leader, sending a full tablet report...
I20260620 12:14:45.525339 3426 ts_tablet_manager.cc:1428] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
W20260620 12:14:45.631688 3139 tablet.cc:2369] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:14:45.642988 3273 tablet.cc:2369] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:14:45.676115 3407 tablet.cc:2369] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:14:45.847018 3430 raft_consensus.cc:482] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:45.847285 3430 raft_consensus.cc:504] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.848685 3430 leader_election.cc:288] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 352500838ccb43ed81b904d0b282c3e6 (127.24.141.2:35169), 8c20809db725438db1323f78abf9cb93 (127.24.141.1:40313)
I20260620 12:14:45.851001 3432 raft_consensus.cc:482] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:14:45.851369 3432 raft_consensus.cc:504] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.853976 3432 leader_election.cc:288] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818ae906e8b3443d8aee6a383ae9f0dc (127.24.141.3:34781), 8c20809db725438db1323f78abf9cb93 (127.24.141.1:40313)
I20260620 12:14:45.855062 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.849003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:35169 (local address 127.24.141.3:33941)
0620 12:14:45.849323 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.849348 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.849482 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.849803 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.849808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.849846 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:45.849998 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.850005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.851379 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.851382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.853226 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.853246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.854544 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.854548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.854670 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.854697 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.854772 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.854885 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 12:14:45.855562 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.849247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:33941 (local address 127.24.141.2:35169)
0620 12:14:45.849447 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.849452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.849469 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:14:45.849603 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.849606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.849657 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.849729 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:45.850186 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.851072 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.853476 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.854149 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.854284 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.854354 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.855329 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.855434 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 12:14:45.855937 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.849506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:53365 (local address 127.24.141.1:40313)
0620 12:14:45.849698 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.849703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.850102 (+ 399us) server_negotiation.cc:407] Connection header received
0620 12:14:45.850459 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.850464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.850533 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.850623 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:45.851277 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.852216 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.854346 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.855013 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.855117 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.855173 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.855683 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.855801 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:14:45.856405 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.849472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:40313 (local address 127.24.141.3:53365)
0620 12:14:45.849957 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.849972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.850093 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.850739 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.850743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.850770 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:45.850932 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.850939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.852399 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.852402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.854161 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.854171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.855298 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.855302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.855414 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.855553 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.856131 (+ 578us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.856230 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":79,"thread_start_us":161,"threads_started":1}
I20260620 12:14:45.856328 3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d22be1b7f1c4d1697c5449cb041c2a0" candidate_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352500838ccb43ed81b904d0b282c3e6" is_pre_election: true
I20260620 12:14:45.856655 3225 raft_consensus.cc:2454] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818ae906e8b3443d8aee6a383ae9f0dc in term 0.
I20260620 12:14:45.857501 3294 leader_election.cc:302] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352500838ccb43ed81b904d0b282c3e6, 818ae906e8b3443d8aee6a383ae9f0dc; no voters:
I20260620 12:14:45.857336 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d22be1b7f1c4d1697c5449cb041c2a0" candidate_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c20809db725438db1323f78abf9cb93" is_pre_election: true
I20260620 12:14:45.858115 3092 raft_consensus.cc:2454] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818ae906e8b3443d8aee6a383ae9f0dc in term 0.
I20260620 12:14:45.857992 3430 raft_consensus.cc:2781] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:14:45.858260 3430 raft_consensus.cc:482] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:14:45.858335 3430 raft_consensus.cc:3037] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:45.859107 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.853788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:47109 (local address 127.24.141.3:34781)
0620 12:14:45.854482 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.854487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.854670 (+ 183us) server_negotiation.cc:407] Connection header received
0620 12:14:45.854890 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.854894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.854950 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.854999 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:45.855426 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.856188 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.857701 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.858310 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.858409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.858462 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.858834 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.858945 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":592}
I20260620 12:14:45.859575 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.853644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34781 (local address 127.24.141.2:47109)
0620 12:14:45.854571 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.854589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.854761 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.855142 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.855146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.855167 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:45.855311 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.855316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.856322 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.856325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.857555 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.857567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.858562 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.858565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.858684 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.858712 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.859316 (+ 604us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.859415 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":145,"threads_started":1}
I20260620 12:14:45.859797 3430 raft_consensus.cc:504] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.860303 3430 leader_election.cc:288] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [CANDIDATE]: Term 1 election: Requested vote from peers 352500838ccb43ed81b904d0b282c3e6 (127.24.141.2:35169), 8c20809db725438db1323f78abf9cb93 (127.24.141.1:40313)
I20260620 12:14:45.860330 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d22be1b7f1c4d1697c5449cb041c2a0" candidate_uuid: "352500838ccb43ed81b904d0b282c3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" is_pre_election: true
I20260620 12:14:45.860725 3360 raft_consensus.cc:2379] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 352500838ccb43ed81b904d0b282c3e6 in current term 1: Already voted for candidate 818ae906e8b3443d8aee6a383ae9f0dc in this term.
I20260620 12:14:45.860742 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d22be1b7f1c4d1697c5449cb041c2a0" candidate_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c20809db725438db1323f78abf9cb93"
I20260620 12:14:45.860807 3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d22be1b7f1c4d1697c5449cb041c2a0" candidate_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352500838ccb43ed81b904d0b282c3e6"
I20260620 12:14:45.860945 3225 raft_consensus.cc:3037] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:45.860939 3092 raft_consensus.cc:3037] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:14:45.861706 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.854181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:40313 (local address 127.24.141.2:60849)
0620 12:14:45.855621 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:45.855637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:14:45.855753 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:14:45.856130 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:14:45.856133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:14:45.856151 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:14:45.856274 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.856280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.858702 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.858705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.859821 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:14:45.859832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.861144 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.861151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:14:45.861280 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.861317 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 12:14:45.861398 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:14:45.861517 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1368,"thread_start_us":99,"threads_started":1}
I20260620 12:14:45.861982 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.854481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:60849 (local address 127.24.141.1:40313)
0620 12:14:45.854932 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.854943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.855833 (+ 890us) server_negotiation.cc:407] Connection header received
0620 12:14:45.855881 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.855884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.855938 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.856018 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:14:45.856415 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.858570 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.859993 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.860730 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.860932 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.860997 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.861708 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.861830 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":137,"threads_started":1}
I20260620 12:14:45.862496 3091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d22be1b7f1c4d1697c5449cb041c2a0" candidate_uuid: "352500838ccb43ed81b904d0b282c3e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c20809db725438db1323f78abf9cb93" is_pre_election: true
I20260620 12:14:45.862673 3092 raft_consensus.cc:2454] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818ae906e8b3443d8aee6a383ae9f0dc in term 1.
I20260620 12:14:45.862673 3225 raft_consensus.cc:2454] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818ae906e8b3443d8aee6a383ae9f0dc in term 1.
I20260620 12:14:45.863083 3294 leader_election.cc:302] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352500838ccb43ed81b904d0b282c3e6, 818ae906e8b3443d8aee6a383ae9f0dc; no voters:
I20260620 12:14:45.863219 3159 leader_election.cc:302] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 352500838ccb43ed81b904d0b282c3e6; no voters: 818ae906e8b3443d8aee6a383ae9f0dc, 8c20809db725438db1323f78abf9cb93
I20260620 12:14:45.863392 3430 raft_consensus.cc:2781] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:14:45.863611 3432 raft_consensus.cc:2726] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 12:14:45.863662 3430 raft_consensus.cc:686] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 1 LEADER]: Becoming Leader. State: Replica: 818ae906e8b3443d8aee6a383ae9f0dc, State: Running, Role: LEADER
I20260620 12:14:45.864032 3430 consensus_queue.cc:237] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } }
I20260620 12:14:45.867712 2947 catalog_manager.cc:5495] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 818ae906e8b3443d8aee6a383ae9f0dc (127.24.141.3). New cstate: current_term: 1 leader_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818ae906e8b3443d8aee6a383ae9f0dc" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 } health_report { overall_health: UNKNOWN } } }
I20260620 12:14:45.954618 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:45.949814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.24.141.3:34781)
0620 12:14:45.950029 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 12:14:45.950033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:14:45.950054 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:14:45.950151 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:14:45.950154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:14:45.950222 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:14:45.950365 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:14:45.950766 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.951391 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.952591 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:14:45.952901 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:14:45.953043 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:14:45.953603 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:14:45.954149 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:14:45.954285 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:14:45.954392 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 12:14:45.954471 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 12:14:45.962996 3225 raft_consensus.cc:1261] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 1 FOLLOWER]: Refusing update from remote peer 818ae906e8b3443d8aee6a383ae9f0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:45.962939 3091 raft_consensus.cc:1261] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 1 FOLLOWER]: Refusing update from remote peer 818ae906e8b3443d8aee6a383ae9f0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:14:45.964128 3437 consensus_queue.cc:1035] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "352500838ccb43ed81b904d0b282c3e6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:45.964414 3430 consensus_queue.cc:1035] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c20809db725438db1323f78abf9cb93" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 40313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:14:45.976011 3441 mvcc.cc:205] Tried to move back new op lower bound from 7298898681694117888 to 7298898681301794816. Current Snapshot: MvccSnapshot[applied={T|T < 7298898681694117888}]
I20260620 12:14:45.990450 2942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:45.990655 2942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:14:45.994392 2942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dad43fd98dd54d779d5037a6d498bc4e: Sending DeleteTablet for 3 replicas of tablet 7d22be1b7f1c4d1697c5449cb041c2a0
I20260620 12:14:45.995821 3072 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d22be1b7f1c4d1697c5449cb041c2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:45 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:48842
I20260620 12:14:45.995954 3206 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d22be1b7f1c4d1697c5449cb041c2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:45 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:51060
I20260620 12:14:45.996407 3450 tablet_replica.cc:330] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93: stopping tablet replica
I20260620 12:14:45.996860 3450 raft_consensus.cc:2229] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:45.997704 3450 raft_consensus.cc:2258] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 8c20809db725438db1323f78abf9cb93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:45.998030 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 3009
I20260620 12:14:46.002218 3451 tablet_replica.cc:330] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6: stopping tablet replica
I20260620 12:14:46.002213 3340 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d22be1b7f1c4d1697c5449cb041c2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:14:45 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:36506
I20260620 12:14:46.002928 3451 raft_consensus.cc:2229] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:14:46.003422 3451 raft_consensus.cc:2258] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 352500838ccb43ed81b904d0b282c3e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 12:14:46.005451 2931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:40313 (error 108)
I20260620 12:14:46.005710 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 3143
I20260620 12:14:46.007272 3452 tablet_replica.cc:330] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: stopping tablet replica
I20260620 12:14:46.007800 3452 raft_consensus.cc:2229] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:14:46.008481 3452 raft_consensus.cc:2258] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:14:46.010272 3452 ts_tablet_manager.cc:1905] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:14:46.012758 3452 ts_tablet_manager.cc:1918] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:14:46.012881 3452 log.cc:1196] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/7d22be1b7f1c4d1697c5449cb041c2a0
I20260620 12:14:46.013211 3452 ts_tablet_manager.cc:1939] T 7d22be1b7f1c4d1697c5449cb041c2a0 P 818ae906e8b3443d8aee6a383ae9f0dc: Deleting consensus metadata
I20260620 12:14:46.014590 2934 catalog_manager.cc:4841] TS 818ae906e8b3443d8aee6a383ae9f0dc (127.24.141.3:34781): tablet 7d22be1b7f1c4d1697c5449cb041c2a0 (table test-table [id=592c3d51e77243e28df746d3750bec99]) successfully deleted
I20260620 12:14:46.015389 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:46.008268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:40313 (local address 127.0.0.1:48858)
0620 12:14:46.015075 (+ 6807us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:46.015220 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6661,"thread_start_us":100,"threads_started":1}
I20260620 12:14:46.015708 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:14:46.015414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:35169 (local address 127.0.0.1:51062)
0620 12:14:46.015592 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 12:14:46.015643 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.2:35169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
W20260620 12:14:46.015691 2931 catalog_manager.cc:4584] TS 8c20809db725438db1323f78abf9cb93 (127.24.141.1:40313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d22be1b7f1c4d1697c5449cb041c2a0: Network error: Client connection negotiation failed: client connection to 127.24.141.1:40313: connect: Connection refused (error 111)
I20260620 12:14:46.015933 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 3277
I20260620 12:14:46.023020 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 2915
2026-06-20T12:14:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2795 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:14:46 dist-test-slave-rh5v krb5kdc[3458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:14:46 dist-test-slave-rh5v krb5kdc[3458](info): set up 2 sockets
Jun 20 12:14:46 dist-test-slave-rh5v krb5kdc[3458](info): commencing operation
krb5kdc: starting...
W20260620 12:14:47.002694 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:14:47 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:14:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/rangeradmin_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/rangeradmin_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/rangerlookup_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/rangerlookup_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
I20260620 12:14:47.196552 25140 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task23Ce8q/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/postgres -l logfile start
2026-06-20 12:14:50.015 UTC [3488] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 12:14:50.015 UTC [3488] LOG: listening on IPv4 address "127.24.141.20", port 59823
2026-06-20 12:14:50.021 UTC [3488] LOG: listening on Unix socket "/tmp/.s.PGSQL.59823"
2026-06-20 12:14:50.046 UTC [3492] LOG: database system was shut down at 2026-06-20 12:14:48 UTC
2026-06-20 12:14:50.050 UTC [3488] LOG: database system is ready to accept connections
I20260620 12:14:50.914249 25140 mini_postgres.cc:96] Postgres bound to 59823
2026-06-20 12:14:50.922 UTC [3500] FATAL: database "slave" does not exist
127.24.141.20:59823 - accepting connections
I20260620 12:14:50.923067 25140 mini_ranger.cc:161] Starting Ranger...
I20260620 12:14:50.941031 25140 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 12:14:51.487599 25140 mini_ranger.cc:87] Created ranger Postgres database
I20260620 12:14:51.487730 25140 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-admin
2026-06-20 12:14:51,824 [I] DB FLAVOR :POSTGRES
2026-06-20 12:14:51,825 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:14:51,825 [I] Checking connection..
2026-06-20 12:14:51,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 12:14:52,124 [I] Checking connection passed.
2026-06-20 12:14:52,124 [I] --------- Verifying version history table ---------
2026-06-20 12:14:52,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:14:52,436 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:14:52,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:14:52,731 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:14:52,732 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 12:14:52,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 12:14:53.026 UTC [3565] WARNING: there is no transaction in progress
2026-06-20 12:14:53,033 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 12:14:53,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:14:53,333 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 12:14:53,334 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:14:53,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 12:14:53,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 12:14:53,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rh5v.c.gcp-upstream.internal','N') ;"
2026-06-20 12:14:54,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:14:54,467 [I] Table x_portal_user does not exist in database ranger
2026-06-20 12:14:54,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:14:54,750 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 12:14:54,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:14:55,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:14:55,339 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 12:14:55,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 12:14:56.553 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.572 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.591 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.611 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.629 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.652 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.731 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.739 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.757 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.770 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.782 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.800 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.814 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.823 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.832 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:56.841 UTC [3673] WARNING: there is no transaction in progress
2026-06-20 12:14:57,657 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 12:14:57,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 12:14:57,918 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 12:14:57,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:14:58,189 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 12:14:58,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:14:58,453 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 12:14:58,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:14:58,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:14:59,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rh5v.c.gcp-upstream.internal';"
2026-06-20 12:14:59,328 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 12:14:59,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 12:14:59,609 [I] DB_PATCHES have already been applied
I20260620 12:14:59.615945 25140 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-admin:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task23Ce8q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 12:14:59.616036 25140 mini_ranger.cc:193] Using host: 127.24.141.20
I20260620 12:14:59.620972 25140 mini_ranger.cc:227] Ranger admin URL: http://127.24.141.20:34023
Jun 20, 2026 12:14:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:14:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:15:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:15:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:15:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:15:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:15:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 12:15:01.763789 25140 mini_ranger.cc:160] Time spent starting Ranger: real 10.841s user 0.010s sys 0.006s
Jun 20, 2026 12:15:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:15:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 12:15:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:15:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:15:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 12:15:28.366813 25140 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/rangerkms_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/rangerkms_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.141.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.141.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 12:15:28 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957728, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 12:15:28.463790 25140 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 12:15:28.478267 25140 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 12:15:29.299682 25140 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 12:15:29.299801 25140 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms
I20260620 12:15:29.299928 25140 mini_ranger_kms.cc:227] Using postgres at 127.24.141.20:59823
2026-06-20 12:15:29,627 [I] DB FLAVOR :POSTGRES
2026-06-20 12:15:29,627 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:15:29,628 [I] Checking connection
2026-06-20 12:15:29,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 12:15:29,921 [I] connection success
2026-06-20 12:15:29,921 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 12:15:29,922 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 12:15:29,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 12:15:30,238 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 12:15:30,238 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:15:30,238 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 12:15:30,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:59823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 12:15:30,556 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 12:15:31.176870 25140 mini_ranger_kms.cc:310] Created kms service
I20260620 12:15:31.495534 25140 mini_ranger_kms.cc:326] Created kudu user
I20260620 12:15:31.570124 25140 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 12:15:31 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957728, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:15:32.091663 25140 mini_ranger_kms.cc:384] Added ranger policy
I20260620 12:15:32.091849 25140 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms:/tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 12:15:32.091871 25140 mini_ranger_kms.cc:240] Using host: 127.24.141.20
I20260620 12:15:32.096118 25140 mini_ranger_kms.cc:276] Ranger KMS PID: 3855
I20260620 12:15:32.096304 25140 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.141.20:53329
Jun 20, 2026 12:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 12:15:34.282030 25140 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.818s user 0.005s sys 0.301s
I20260620 12:15:34.282258 25140 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 12:15:34.282393 25140 mini_ranger_kms.cc:401] 127.24.141.20:53329/kms/v1/keys
Jun 20 12:15:34 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781957734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.141.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 12:15:35 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.141.20@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
Jun 20 12:15:35 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.141.20@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
Jun 20 12:15:38 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:15:38.589560 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44273
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:44273
--ranger_config_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:53329/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:15:38.736930 3905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:15:38.737315 3905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:15:38.737365 3905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:15:38.742528 3905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:15:38.742632 3905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:15:38.742653 3905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:15:38.742671 3905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:15:38.742686 3905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:15:38.749138 3905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.141.20:53329/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:44273
--ranger_config_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:15:38.750432 3905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:15:38.751623 3905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:15:38.763438 3911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:15:38.763449 3910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:15:38.763528 3913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:15:38.764516 3905 server_base.cc:1034] running on GCE node
I20260620 12:15:38.765542 3905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:15:38.767493 3905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:15:38.768795 3905 hybrid_clock.cc:648] HybridClock initialized: now 1781957738768745 us; error 77 us; skew 500 ppm
Jun 20 12:15:38 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957738, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:15:38.779539 3905 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:15:38.780824 3905 webserver.cc:460] Webserver started at http://127.24.141.62:44159/ using document root <none> and password file <none>
I20260620 12:15:38.781508 3905 fs_manager.cc:359] Metadata directory not provided
I20260620 12:15:38.781572 3905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:15:38.781813 3905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:15:38 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957738, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:15:38.855723 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "028289d7dd424e3e92ef94cd267ce2ce"
format_stamp: "Formatted at 2026-06-20 12:15:38 on dist-test-slave-rh5v"
server_key: "b4651ebfa2c9edb1fc472edec5df8858"
server_key_iv: "ff3c60d731af54f883af84ed18c8ba14"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:38.856740 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "028289d7dd424e3e92ef94cd267ce2ce"
format_stamp: "Formatted at 2026-06-20 12:15:38 on dist-test-slave-rh5v"
server_key: "b4651ebfa2c9edb1fc472edec5df8858"
server_key_iv: "ff3c60d731af54f883af84ed18c8ba14"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:38.863550 3905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 12:15:38.881239 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:38.882931 3905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:15:38.883239 3905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "028289d7dd424e3e92ef94cd267ce2ce"
format_stamp: "Formatted at 2026-06-20 12:15:38 on dist-test-slave-rh5v"
server_key: "b4651ebfa2c9edb1fc472edec5df8858"
server_key_iv: "ff3c60d731af54f883af84ed18c8ba14"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:38.883407 3905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:15:38.897782 3905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:15:38.901105 3905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:15:38.901322 3905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:15:38.911211 3905 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:44273
I20260620 12:15:38.911352 3972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:44273 every 8 connection(s)
I20260620 12:15:38.912468 3905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:15:38.916103 3973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:15:38.922256 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 3905
I20260620 12:15:38.922441 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:15:38.922807 25140 external_mini_cluster.cc:1428] Setting key 9e4f349588e3c79bd66d04f4eff5a272
I20260620 12:15:38.923308 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce: Bootstrap starting.
I20260620 12:15:38.926738 3973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce: Neither blocks nor log segments found. Creating new log.
I20260620 12:15:38.927810 3973 log.cc:824] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce: Log is configured to *not* fsync() on all Append() calls
I20260620 12:15:38.930576 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce: No bootstrap required, opened a new log
Jun 20 12:15:38 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:15:38.935707 3973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028289d7dd424e3e92ef94cd267ce2ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44273 } }
I20260620 12:15:38.936075 3973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:15:38.936136 3973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028289d7dd424e3e92ef94cd267ce2ce, State: Initialized, Role: FOLLOWER
I20260620 12:15:38.936805 3973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028289d7dd424e3e92ef94cd267ce2ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44273 } }
I20260620 12:15:38.936985 3973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:15:38.937103 3973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:15:38.937233 3973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:15:38.938465 3973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028289d7dd424e3e92ef94cd267ce2ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44273 } }
I20260620 12:15:38.938889 3973 leader_election.cc:302] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 028289d7dd424e3e92ef94cd267ce2ce; no voters:
I20260620 12:15:38.939275 3973 leader_election.cc:288] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:15:38.939392 3978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:15:38.939643 3978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [term 1 LEADER]: Becoming Leader. State: Replica: 028289d7dd424e3e92ef94cd267ce2ce, State: Running, Role: LEADER
I20260620 12:15:38.940116 3978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028289d7dd424e3e92ef94cd267ce2ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44273 } }
I20260620 12:15:38.940862 3973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:15:38.942293 3979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "028289d7dd424e3e92ef94cd267ce2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028289d7dd424e3e92ef94cd267ce2ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44273 } } }
I20260620 12:15:38.942530 3979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [sys.catalog]: This master's current role is: LEADER
I20260620 12:15:38.942353 3980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 028289d7dd424e3e92ef94cd267ce2ce. Latest consensus state: current_term: 1 leader_uuid: "028289d7dd424e3e92ef94cd267ce2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028289d7dd424e3e92ef94cd267ce2ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44273 } } }
I20260620 12:15:38.943043 3980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce [sys.catalog]: This master's current role is: LEADER
I20260620 12:15:38.943070 3988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:15:38.943502 3973 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:15:38.944185 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:38.925009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.24.141.62:44273)
0620 12:15:38.925735 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:38.925756 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:38.925807 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:15:38.926753 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:38.926783 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:38.927317 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:38.927750 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:15:38.928967 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:38.930488 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:38.931999 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:38.932832 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:38.933119 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:38.936995 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:15:38.937006 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:15:38.937037 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:15:38.937135 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:15:38.939328 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:38.940080 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:38.940087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:38.940097 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:38.940249 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:38.940588 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:38.940591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:38.940592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:38.941245 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:15:38.941536 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:38.942331 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 12:15:38.942759 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":238,"threads_started":1}
I20260620 12:15:38.953469 3988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:15:38.963086 3988 catalog_manager.cc:1347] Generated new cluster ID: 5059531bf2e14e659920a687326b2b2e
I20260620 12:15:38.963181 3988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:15:38.984869 3988 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:15:38.986478 3988 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:15:39.002837 3988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce: Generated new TSK 0
I20260620 12:15:39.004055 3988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:15:40.075502 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:53329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44273
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:15:40.311082 4018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:15:40.311604 4018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:15:40.311663 4018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:15:40.316566 4018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:15:40.316680 4018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:15:40.316751 4018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:15:40.323565 4018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.141.20:53329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:15:40.325310 4018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:15:40.327036 4018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:15:40.341488 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:15:40.341434 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:15:40.342115 4018 server_base.cc:1034] running on GCE node
W20260620 12:15:40.341466 4028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:15:40.343019 4018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:15:40.344199 4018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:15:40.345448 4018 hybrid_clock.cc:648] HybridClock initialized: now 1781957740345436 us; error 73 us; skew 500 ppm
Jun 20 12:15:40 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:15:40.350916 4018 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:15:40.352499 4018 webserver.cc:460] Webserver started at http://127.24.141.1:34067/ using document root <none> and password file <none>
I20260620 12:15:40.353518 4018 fs_manager.cc:359] Metadata directory not provided
I20260620 12:15:40.353644 4018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:15:40.354051 4018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:15:40 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:15:40.378268 4018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "ae7c65d01c5c497bbd2558b76b219dea"
format_stamp: "Formatted at 2026-06-20 12:15:40 on dist-test-slave-rh5v"
server_key: "c9330d0f1cd43b05a4d7923c4418748e"
server_key_iv: "6f81238821760114c0e0d6bf23447221"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:40.379115 4018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "ae7c65d01c5c497bbd2558b76b219dea"
format_stamp: "Formatted at 2026-06-20 12:15:40 on dist-test-slave-rh5v"
server_key: "c9330d0f1cd43b05a4d7923c4418748e"
server_key_iv: "6f81238821760114c0e0d6bf23447221"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:40.385133 4018 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 12:15:40.421753 4035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:40.423949 4018 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260620 12:15:40.424155 4018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "ae7c65d01c5c497bbd2558b76b219dea"
format_stamp: "Formatted at 2026-06-20 12:15:40 on dist-test-slave-rh5v"
server_key: "c9330d0f1cd43b05a4d7923c4418748e"
server_key_iv: "6f81238821760114c0e0d6bf23447221"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:40.424321 4018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:15:40.453754 4018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:15:40.457588 4018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:15:40.457824 4018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:15:40.458773 4018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:15:40.460059 4018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:15:40.460124 4018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:40.460186 4018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:15:40.460214 4018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:40.491816 4018 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:35677
I20260620 12:15:40.493764 4018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:15:40.495827 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:35677 every 8 connection(s)
I20260620 12:15:40.500263 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 4018
I20260620 12:15:40.500432 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:15:40.500836 25140 external_mini_cluster.cc:1428] Setting key e319272536fe112f8efdb8166e325ea4
Jun 20 12:15:40 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:15:40.517369 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:40.497680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:54721 (local address 127.24.141.62:44273)
0620 12:15:40.498789 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:40.498796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:40.498828 (+ 32us) server_negotiation.cc:407] Connection header received
0620 12:15:40.500074 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:40.500079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:40.500160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:40.500262 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:15:40.502426 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.503382 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.505448 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.505862 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.506001 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:40.511152 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:15:40.511158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:15:40.511162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:15:40.511210 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:15:40.513794 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:40.514778 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:40.514783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:40.514785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:40.514910 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:40.515441 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:40.515445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:40.515448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:40.515723 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:15:40.515811 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:40.516830 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0620 12:15:40.517103 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":83,"threads_started":1}
I20260620 12:15:40.518437 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:40.497258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44273 (local address 127.24.141.1:54721)
0620 12:15:40.498049 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:40.498133 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:40.499565 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:40.500701 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:40.500725 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:40.501251 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:15:40.502208 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:40.502231 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.503573 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.503577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:40.505269 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:40.505284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.506197 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.506206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:40.506691 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:40.508041 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:15:40.508070 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:15:40.510662 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:15:40.514042 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:15:40.514052 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:15:40.514078 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:15:40.514617 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:15:40.515043 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:15:40.515046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:15:40.515048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:15:40.515313 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:15:40.515890 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:15:40.515898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:15:40.516308 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 12:15:40.516549 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 12:15:40.516982 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":235,"threads_started":1}
I20260620 12:15:40.520723 4151 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44273
I20260620 12:15:40.521256 4151 heartbeater.cc:464] Registering TS with master...
I20260620 12:15:40.522167 4151 heartbeater.cc:511] Master 127.24.141.62:44273 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:15:40.525031 3934 ts_manager.cc:194] Registered new tserver with Master: ae7c65d01c5c497bbd2558b76b219dea (127.24.141.1:35677)
I20260620 12:15:40.526953 3934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:54721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:15:40.544245 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:53329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44273
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:15:40 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 12:15:40.705354 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:15:40.705744 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:15:40.705812 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:15:40.710803 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:15:40.710911 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:15:40.711014 4157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:15:40.717785 4157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.141.20:53329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:15:40.719439 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:15:40.720865 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:15:40.735412 4163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:15:40.735829 4164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:15:40.735417 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:15:40.736143 4157 server_base.cc:1034] running on GCE node
I20260620 12:15:40.736681 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:15:40.737586 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:15:40.738770 4157 hybrid_clock.cc:648] HybridClock initialized: now 1781957740738737 us; error 47 us; skew 500 ppm
Jun 20 12:15:40 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:15:40.742373 4157 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:15:40.743718 4157 webserver.cc:460] Webserver started at http://127.24.141.2:44007/ using document root <none> and password file <none>
I20260620 12:15:40.744436 4157 fs_manager.cc:359] Metadata directory not provided
I20260620 12:15:40.744516 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:15:40.744786 4157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:15:40 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:15:40.775537 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "819b5dbf536b42d9b01ffa4f5c896f07"
format_stamp: "Formatted at 2026-06-20 12:15:40 on dist-test-slave-rh5v"
server_key: "70e2755aac6b284401b556764fab784f"
server_key_iv: "c4cccb0f51458f1274866fef0ad68371"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:40.776911 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "819b5dbf536b42d9b01ffa4f5c896f07"
format_stamp: "Formatted at 2026-06-20 12:15:40 on dist-test-slave-rh5v"
server_key: "70e2755aac6b284401b556764fab784f"
server_key_iv: "c4cccb0f51458f1274866fef0ad68371"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:40.794588 4157 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.001s sys 0.004s
I20260620 12:15:40.833698 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:40.836244 4157 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260620 12:15:40.836539 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "819b5dbf536b42d9b01ffa4f5c896f07"
format_stamp: "Formatted at 2026-06-20 12:15:40 on dist-test-slave-rh5v"
server_key: "70e2755aac6b284401b556764fab784f"
server_key_iv: "c4cccb0f51458f1274866fef0ad68371"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:40.836781 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:15:40.855651 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:15:40.860594 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:15:40.860934 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:15:40.861924 4157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:15:40.863407 4157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:15:40.863528 4157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 12:15:40.863641 4157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:15:40.863713 4157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:40.879842 4157 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:38805
I20260620 12:15:40.882054 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:15:40.886032 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:38805 every 8 connection(s)
I20260620 12:15:40.889333 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 4157
I20260620 12:15:40.889482 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:15:40.889770 25140 external_mini_cluster.cc:1428] Setting key 5ac85f708641026e2b9f7c5c65815265
Jun 20 12:15:40 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:15:40.912599 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:40.889197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:47537 (local address 127.24.141.62:44273)
0620 12:15:40.889387 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:40.889393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:40.892196 (+ 2803us) server_negotiation.cc:407] Connection header received
0620 12:15:40.893678 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:40.893683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:40.893766 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:40.893826 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:15:40.895795 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.896677 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.898741 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.899073 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.899171 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:40.906741 (+ 7570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:15:40.906746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:15:40.906750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:15:40.906789 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:15:40.909201 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:40.910236 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:40.910241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:40.910243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:40.910369 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:40.910929 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:40.910934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:40.910936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:40.911269 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:15:40.911370 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:40.912223 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 12:15:40.912412 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:15:40.914872 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:40.889858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44273 (local address 127.24.141.2:47537)
0620 12:15:40.891984 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:40.892055 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:40.893373 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:40.894362 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:40.894373 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:40.894798 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:15:40.895573 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:40.895595 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.896871 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.896875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:40.898541 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:40.898560 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:40.899456 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:40.899460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:40.899693 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:40.904349 (+ 4656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:15:40.904381 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:15:40.906432 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:15:40.909491 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:15:40.909501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:15:40.909530 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:15:40.910037 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:15:40.910521 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:15:40.910525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:15:40.910527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:15:40.910785 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:15:40.911479 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:15:40.911488 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:15:40.911884 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 12:15:40.913146 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0620 12:15:40.913669 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1851,"thread_start_us":329,"threads_started":1}
I20260620 12:15:40.916927 4287 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44273
I20260620 12:15:40.917292 4287 heartbeater.cc:464] Registering TS with master...
I20260620 12:15:40.918027 4287 heartbeater.cc:511] Master 127.24.141.62:44273 requested a full tablet report, sending...
I20260620 12:15:40.919564 3934 ts_manager.cc:194] Registered new tserver with Master: 819b5dbf536b42d9b01ffa4f5c896f07 (127.24.141.2:38805)
I20260620 12:15:40.920428 3934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:47537
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:15:40.944532 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:53329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44273
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:15:41.140156 4292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:15:41.140705 4292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:15:41.140837 4292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:15:41.148573 4292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:15:41.148772 4292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:15:41.148908 4292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:15:41.157506 4292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.141.20:53329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:44273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:15:41.159592 4292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:15:41.161500 4292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 12:15:41 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
W20260620 12:15:41.177160 4297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:15:41.177160 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:15:41.177428 4298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:15:41.179421 4292 server_base.cc:1034] running on GCE node
I20260620 12:15:41.180097 4292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:15:41.181032 4292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:15:41.184373 4292 hybrid_clock.cc:648] HybridClock initialized: now 1781957741184280 us; error 113 us; skew 500 ppm
Jun 20 12:15:41 dist-test-slave-rh5v krb5kdc[3458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957741, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:15:41.190477 4292 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:15:41.191906 4292 webserver.cc:460] Webserver started at http://127.24.141.3:36257/ using document root <none> and password file <none>
I20260620 12:15:41.192828 4292 fs_manager.cc:359] Metadata directory not provided
I20260620 12:15:41.192993 4292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:15:41.193367 4292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:15:41 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957741, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:15:41.216915 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "08fad48107b7400381f9be11e6402dea"
format_stamp: "Formatted at 2026-06-20 12:15:41 on dist-test-slave-rh5v"
server_key: "13717efde0f30df55610569ccdeec1dc"
server_key_iv: "b89a7278479cbaa8c4c35b6599370141"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:41.219017 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "08fad48107b7400381f9be11e6402dea"
format_stamp: "Formatted at 2026-06-20 12:15:41 on dist-test-slave-rh5v"
server_key: "13717efde0f30df55610569ccdeec1dc"
server_key_iv: "b89a7278479cbaa8c4c35b6599370141"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:41.226714 4292 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 12:15:41.269798 4307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:41.272397 4292 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 12:15:41.272632 4292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "08fad48107b7400381f9be11e6402dea"
format_stamp: "Formatted at 2026-06-20 12:15:41 on dist-test-slave-rh5v"
server_key: "13717efde0f30df55610569ccdeec1dc"
server_key_iv: "b89a7278479cbaa8c4c35b6599370141"
server_key_version: "kuduclusterkey@0"
I20260620 12:15:41.272868 4292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:15:41.294095 4292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:15:41.298605 4292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:15:41.298949 4292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:15:41.299929 4292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:15:41.301359 4292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:15:41.301481 4292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:41.301586 4292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:15:41.301646 4292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:15:41.321653 4292 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:41581
I20260620 12:15:41.323434 4292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:15:41.326181 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 4292
I20260620 12:15:41.326335 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:15:41.326655 25140 external_mini_cluster.cc:1428] Setting key 395b54d7cad927df7c3a7cb6e7c4ebf6
I20260620 12:15:41.324255 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:41581 every 8 connection(s)
Jun 20 12:15:41 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957741, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:15:41.353051 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.330534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:40041 (local address 127.24.141.62:44273)
0620 12:15:41.330694 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.330699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.334073 (+ 3374us) server_negotiation.cc:407] Connection header received
0620 12:15:41.335856 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.335861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.335952 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.336038 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:15:41.338827 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.339711 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.341772 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.342165 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.342254 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.346489 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:15:41.346496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:15:41.346499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:15:41.346552 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:15:41.348803 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:41.350054 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:41.350060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:41.350062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:41.350213 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:41.350798 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:41.350802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:41.350804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:41.351142 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:15:41.351246 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.352567 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.352816 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:15:41.355890 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.328946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44273 (local address 127.24.141.3:40041)
0620 12:15:41.333770 (+ 4824us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.333876 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:41.335451 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:41.337036 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:41.337051 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:41.337595 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:15:41.338572 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.338597 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.339923 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.339927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.341585 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.341602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.342547 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.342551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.342821 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.343731 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:15:41.343758 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:15:41.346070 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:15:41.349139 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:15:41.349152 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:15:41.349188 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:15:41.349824 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:15:41.350360 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:15:41.350363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:15:41.350365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:15:41.350663 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:15:41.351342 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:15:41.351353 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:15:41.352087 (+ 734us) client_negotiation.cc:769] Sending connection context
0620 12:15:41.353769 (+ 1682us) client_negotiation.cc:240] Negotiation successful
0620 12:15:41.354373 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4345,"thread_start_us":207,"threads_started":1}
I20260620 12:15:41.358232 4421 heartbeater.cc:347] Connected to a master server at 127.24.141.62:44273
I20260620 12:15:41.358623 4421 heartbeater.cc:464] Registering TS with master...
I20260620 12:15:41.359519 4421 heartbeater.cc:511] Master 127.24.141.62:44273 requested a full tablet report, sending...
I20260620 12:15:41.361585 3934 ts_manager.cc:194] Registered new tserver with Master: 08fad48107b7400381f9be11e6402dea (127.24.141.3:41581)
I20260620 12:15:41.362637 3934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:40041
I20260620 12:15:41.368258 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:15:41.387655 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.374803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.24.141.62:44273)
0620 12:15:41.374983 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.374988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.375982 (+ 994us) server_negotiation.cc:407] Connection header received
0620 12:15:41.376247 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.376251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.376316 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.376373 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:15:41.377755 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.378643 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.380244 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.380601 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.380695 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.383182 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:15:41.383188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:15:41.383191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:15:41.383242 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:15:41.385188 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:41.385872 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:41.385877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:41.385879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:41.385986 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:15:41.386326 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:15:41.386330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:15:41.386332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:15:41.386636 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:15:41.386761 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.387271 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.387448 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:15:41.393043 3934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:15:41.399241 3934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:15:41.456224 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.424519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:41581 (local address 127.0.0.1:39228)
0620 12:15:41.428717 (+ 4198us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.428731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:41.428835 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:41.429828 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:41.429836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:41.429864 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:15:41.430175 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.430185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.446924 (+ 16739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.446929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.448269 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.448288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.455397 (+ 7109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.455402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.455616 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.455670 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 12:15:41.455814 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 12:15:41.456008 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4131,"thread_start_us":119,"threads_started":1}
I20260620 12:15:41.457993 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.426024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38805 (local address 127.0.0.1:45554)
0620 12:15:41.432110 (+ 6086us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.432131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:41.434882 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:41.438355 (+ 3473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:41.438360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:41.438391 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:15:41.438561 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.438570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.443694 (+ 5124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.443698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.448693 (+ 4995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.448702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.457481 (+ 8779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.457486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.457616 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.457644 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:15:41.457719 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:15:41.457821 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4321,"thread_start_us":116,"threads_started":1}
I20260620 12:15:41.462178 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.426318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.24.141.2:38805)
0620 12:15:41.433413 (+ 7095us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.433422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.433467 (+ 45us) server_negotiation.cc:407] Connection header received
0620 12:15:41.435199 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.435209 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.437532 (+ 2323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.437900 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:15:41.438744 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.440009 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.448863 (+ 8854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.457094 (+ 8231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.457220 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.457400 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.461734 (+ 4334us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.461953 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6885,"thread_start_us":128,"threads_started":1}
I20260620 12:15:41.467625 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.421014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:35677 (local address 127.0.0.1:44342)
0620 12:15:41.427871 (+ 6857us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.427971 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:41.428140 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:41.440769 (+ 12629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:41.440773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:41.440806 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:15:41.440949 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.440957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.451924 (+ 10967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.451928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.453378 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.453397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.466888 (+ 13491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.466894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.467073 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.467114 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 12:15:41.467225 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 12:15:41.467359 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6578,"thread_start_us":234,"threads_started":1}
I20260620 12:15:41.468473 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.425042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.24.141.3:41581)
0620 12:15:41.428555 (+ 3513us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.428571 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.428931 (+ 360us) server_negotiation.cc:407] Connection header received
0620 12:15:41.429039 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.429046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.429407 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.429603 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:15:41.441166 (+ 11563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.446695 (+ 5529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.453684 (+ 6989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.454775 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.454918 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.455145 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.465558 (+ 10413us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.468280 (+ 2722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3350,"thread_start_us":139,"threads_started":1}
I20260620 12:15:41.472190 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.422260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.24.141.1:35677)
0620 12:15:41.432650 (+ 10390us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.432663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.432704 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:15:41.432822 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.432830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.440233 (+ 7403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.440495 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:15:41.445257 (+ 4762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.451741 (+ 6484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.458138 (+ 6397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.466262 (+ 8124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.466473 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.466743 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.471713 (+ 4970us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.471960 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9583,"thread_start_us":163,"threads_started":1}
I20260620 12:15:41.476631 4078 tablet_service.cc:1467] Processing CreateTablet for tablet adac9fd451ee40cd84acb12a728da833 (DEFAULT_TABLE table=test-table [id=4e22dcea4a424117b52b902ed43e3a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:15:41.476641 4339 tablet_service.cc:1467] Processing CreateTablet for tablet adac9fd451ee40cd84acb12a728da833 (DEFAULT_TABLE table=test-table [id=4e22dcea4a424117b52b902ed43e3a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:15:41.478228 4078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adac9fd451ee40cd84acb12a728da833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:15:41.478228 4339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adac9fd451ee40cd84acb12a728da833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:15:41.485227 4221 tablet_service.cc:1467] Processing CreateTablet for tablet adac9fd451ee40cd84acb12a728da833 (DEFAULT_TABLE table=test-table [id=4e22dcea4a424117b52b902ed43e3a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:15:41.487172 4221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adac9fd451ee40cd84acb12a728da833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:15:41.510087 4440 tablet_bootstrap.cc:492] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: Bootstrap starting.
I20260620 12:15:41.514873 4440 tablet_bootstrap.cc:654] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: Neither blocks nor log segments found. Creating new log.
I20260620 12:15:41.517560 4440 log.cc:824] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: Log is configured to *not* fsync() on all Append() calls
I20260620 12:15:41.518520 4441 tablet_bootstrap.cc:492] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: Bootstrap starting.
I20260620 12:15:41.522351 4441 tablet_bootstrap.cc:654] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: Neither blocks nor log segments found. Creating new log.
I20260620 12:15:41.523502 4442 tablet_bootstrap.cc:492] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea: Bootstrap starting.
I20260620 12:15:41.523957 4441 log.cc:824] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: Log is configured to *not* fsync() on all Append() calls
I20260620 12:15:41.527355 4441 tablet_bootstrap.cc:492] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: No bootstrap required, opened a new log
I20260620 12:15:41.527693 4441 ts_tablet_manager.cc:1397] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 12:15:41.531286 4151 heartbeater.cc:503] Master 127.24.141.62:44273 was elected leader, sending a full tablet report...
I20260620 12:15:41.535214 4440 tablet_bootstrap.cc:492] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: No bootstrap required, opened a new log
I20260620 12:15:41.535516 4440 ts_tablet_manager.cc:1397] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.003s
I20260620 12:15:41.540271 4440 raft_consensus.cc:348] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.540728 4440 raft_consensus.cc:374] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:15:41.540798 4440 raft_consensus.cc:729] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fad48107b7400381f9be11e6402dea, State: Initialized, Role: FOLLOWER
I20260620 12:15:41.541289 4441 raft_consensus.cc:348] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.541817 4441 raft_consensus.cc:374] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:15:41.541908 4441 raft_consensus.cc:729] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819b5dbf536b42d9b01ffa4f5c896f07, State: Initialized, Role: FOLLOWER
I20260620 12:15:41.542660 4441 consensus_queue.cc:260] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.543869 4442 tablet_bootstrap.cc:654] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea: Neither blocks nor log segments found. Creating new log.
I20260620 12:15:41.544545 4441 ts_tablet_manager.cc:1428] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20260620 12:15:41.545079 4287 heartbeater.cc:503] Master 127.24.141.62:44273 was elected leader, sending a full tablet report...
I20260620 12:15:41.546770 4442 log.cc:824] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea: Log is configured to *not* fsync() on all Append() calls
I20260620 12:15:41.550410 4440 consensus_queue.cc:260] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.552047 4440 ts_tablet_manager.cc:1428] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20260620 12:15:41.554528 4442 tablet_bootstrap.cc:492] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea: No bootstrap required, opened a new log
I20260620 12:15:41.554806 4442 ts_tablet_manager.cc:1397] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20260620 12:15:41.559334 4442 raft_consensus.cc:348] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.559725 4442 raft_consensus.cc:374] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:15:41.561650 4421 heartbeater.cc:503] Master 127.24.141.62:44273 was elected leader, sending a full tablet report...
I20260620 12:15:41.570600 4442 raft_consensus.cc:729] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7c65d01c5c497bbd2558b76b219dea, State: Initialized, Role: FOLLOWER
I20260620 12:15:41.571626 4442 consensus_queue.cc:260] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.574316 4442 ts_tablet_manager.cc:1428] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
W20260620 12:15:41.577546 4422 tablet.cc:2369] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:15:41.592478 4445 raft_consensus.cc:482] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:15:41.592918 4445 raft_consensus.cc:504] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.594754 4445 leader_election.cc:288] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae7c65d01c5c497bbd2558b76b219dea (127.24.141.1:35677), 08fad48107b7400381f9be11e6402dea (127.24.141.3:41581)
I20260620 12:15:41.601815 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.596357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:48931 (local address 127.24.141.1:35677)
0620 12:15:41.596492 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.596496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.596775 (+ 279us) server_negotiation.cc:407] Connection header received
0620 12:15:41.597005 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.597009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.597099 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.597149 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:15:41.597676 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.598509 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.600251 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.600901 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.601016 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.601123 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.601513 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.601626 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:15:41.602851 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.595085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:35677 (local address 127.24.141.2:48931)
0620 12:15:41.596674 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.596691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:41.596876 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:41.597312 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:41.597315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:41.597354 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:15:41.597539 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.597547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.598678 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.598681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.600073 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.600089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.601247 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.601250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.601354 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.601379 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:15:41.602524 (+ 1145us) client_negotiation.cc:240] Negotiation successful
0620 12:15:41.602647 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"thread_start_us":182,"threads_started":1}
I20260620 12:15:41.603993 4105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac9fd451ee40cd84acb12a728da833" candidate_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7c65d01c5c497bbd2558b76b219dea" is_pre_election: true
I20260620 12:15:41.604481 4105 raft_consensus.cc:2454] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819b5dbf536b42d9b01ffa4f5c896f07 in term 0.
I20260620 12:15:41.605499 4174 leader_election.cc:302] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819b5dbf536b42d9b01ffa4f5c896f07, ae7c65d01c5c497bbd2558b76b219dea; no voters:
I20260620 12:15:41.606176 4445 raft_consensus.cc:2781] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:15:41.606292 4445 raft_consensus.cc:482] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:15:41.606329 4445 raft_consensus.cc:3037] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:15:41.608053 4445 raft_consensus.cc:504] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.608588 4445 leader_election.cc:288] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [CANDIDATE]: Term 1 election: Requested vote from peers ae7c65d01c5c497bbd2558b76b219dea (127.24.141.1:35677), 08fad48107b7400381f9be11e6402dea (127.24.141.3:41581)
I20260620 12:15:41.609653 4105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac9fd451ee40cd84acb12a728da833" candidate_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7c65d01c5c497bbd2558b76b219dea"
I20260620 12:15:41.609838 4105 raft_consensus.cc:3037] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:15:41.611510 4105 raft_consensus.cc:2454] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819b5dbf536b42d9b01ffa4f5c896f07 in term 1.
I20260620 12:15:41.611881 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.596201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:53899 (local address 127.24.141.3:41581)
0620 12:15:41.596351 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.596357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.607185 (+ 10828us) server_negotiation.cc:407] Connection header received
0620 12:15:41.607479 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.607484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.607539 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.607587 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:15:41.608009 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.608798 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.610508 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.611107 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.611215 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.611290 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.611624 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.611715 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:15:41.612360 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.595644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:41581 (local address 127.24.141.2:53899)
0620 12:15:41.607041 (+ 11397us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.607063 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:15:41.607328 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:15:41.607744 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:15:41.607748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:15:41.607767 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:15:41.607878 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.607885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.609103 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.609107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.610316 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:15:41.610331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.611394 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.611398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:15:41.611490 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.611518 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:15:41.612086 (+ 568us) client_negotiation.cc:240] Negotiation successful
0620 12:15:41.612181 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11288,"thread_start_us":102,"threads_started":1}
I20260620 12:15:41.612813 4174 leader_election.cc:302] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819b5dbf536b42d9b01ffa4f5c896f07, ae7c65d01c5c497bbd2558b76b219dea; no voters:
I20260620 12:15:41.614378 4369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac9fd451ee40cd84acb12a728da833" candidate_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fad48107b7400381f9be11e6402dea" is_pre_election: true
I20260620 12:15:41.614836 4369 raft_consensus.cc:2454] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819b5dbf536b42d9b01ffa4f5c896f07 in term 0.
I20260620 12:15:41.615288 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac9fd451ee40cd84acb12a728da833" candidate_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fad48107b7400381f9be11e6402dea"
I20260620 12:15:41.615418 4375 raft_consensus.cc:3037] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:15:41.617779 4375 raft_consensus.cc:2454] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819b5dbf536b42d9b01ffa4f5c896f07 in term 1.
I20260620 12:15:41.619138 4445 raft_consensus.cc:2781] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:15:41.619536 4445 raft_consensus.cc:686] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 1 LEADER]: Becoming Leader. State: Replica: 819b5dbf536b42d9b01ffa4f5c896f07, State: Running, Role: LEADER
I20260620 12:15:41.620033 4445 consensus_queue.cc:237] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } }
I20260620 12:15:41.625497 3934 catalog_manager.cc:5495] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819b5dbf536b42d9b01ffa4f5c896f07 (127.24.141.2). New cstate: current_term: 1 leader_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819b5dbf536b42d9b01ffa4f5c896f07" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38805 } health_report { overall_health: HEALTHY } } }
W20260620 12:15:41.642549 4288 tablet.cc:2369] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:15:41.645961 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.639300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.24.141.2:38805)
0620 12:15:41.639553 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 12:15:41.639559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:15:41.639578 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:15:41.639696 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:15:41.639699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:15:41.639797 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:15:41.639932 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:15:41.640382 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.641256 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.642755 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:15:41.643106 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:15:41.643209 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:15:41.643983 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:15:41.644768 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:15:41.644890 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:15:41.645685 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 12:15:41.645769 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260620 12:15:41.655979 4375 raft_consensus.cc:1261] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 1 FOLLOWER]: Refusing update from remote peer 819b5dbf536b42d9b01ffa4f5c896f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 20 12:15:41 dist-test-slave-rh5v krb5kdc[3458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:15:41.657593 4452 consensus_queue.cc:1035] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08fad48107b7400381f9be11e6402dea" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:15:41.658543 4105 raft_consensus.cc:1261] T adac9fd451ee40cd84acb12a728da833 P ae7c65d01c5c497bbd2558b76b219dea [term 1 FOLLOWER]: Refusing update from remote peer 819b5dbf536b42d9b01ffa4f5c896f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:15:41.663288 4452 consensus_queue.cc:1035] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae7c65d01c5c497bbd2558b76b219dea" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:15:41.689482 4456 mvcc.cc:205] Tried to move back new op lower bound from 7298898909809627136 to 7298898909681508352. Current Snapshot: MvccSnapshot[applied={T|T < 7298898909809627136}]
I20260620 12:15:41.698729 4458 mvcc.cc:205] Tried to move back new op lower bound from 7298898909809627136 to 7298898909681508352. Current Snapshot: MvccSnapshot[applied={T|T < 7298898909809627136}]
I20260620 12:15:41.717500 3934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:15:41.717790 3934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:15:41.721591 3934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 028289d7dd424e3e92ef94cd267ce2ce: Sending DeleteTablet for 3 replicas of tablet adac9fd451ee40cd84acb12a728da833
I20260620 12:15:41.723264 4078 tablet_service.cc:1514] Processing DeleteTablet for tablet adac9fd451ee40cd84acb12a728da833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:15:41 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:44342
I20260620 12:15:41.723999 4339 tablet_service.cc:1514] Processing DeleteTablet for tablet adac9fd451ee40cd84acb12a728da833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:15:41 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:39228
I20260620 12:15:41.725373 4221 tablet_service.cc:1514] Processing DeleteTablet for tablet adac9fd451ee40cd84acb12a728da833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:15:41 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45554
I20260620 12:15:41.726722 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 4018
I20260620 12:15:41.729537 4465 tablet_replica.cc:330] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: stopping tablet replica
I20260620 12:15:41.730073 4465 raft_consensus.cc:2229] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:15:41.730545 4465 raft_consensus.cc:2258] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:15:41.738181 4466 tablet_replica.cc:330] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07: stopping tablet replica
W20260620 12:15:41.740079 3921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:35677 (error 108)
I20260620 12:15:41.741295 4466 raft_consensus.cc:2229] T adac9fd451ee40cd84acb12a728da833 P 819b5dbf536b42d9b01ffa4f5c896f07 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:15:41.741434 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 4157
I20260620 12:15:41.746240 4465 ts_tablet_manager.cc:1905] T adac9fd451ee40cd84acb12a728da833 P 08fad48107b7400381f9be11e6402dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:15:41.753026 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 4292
I20260620 12:15:41.755172 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.754454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:35677 (local address 127.0.0.1:44348)
0620 12:15:41.754945 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.755046 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
I20260620 12:15:41.755424 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.754769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38805 (local address 127.0.0.1:45568)
0620 12:15:41.755337 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.755380 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.2:38805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
W20260620 12:15:41.755887 3924 catalog_manager.cc:4584] TS 819b5dbf536b42d9b01ffa4f5c896f07 (127.24.141.2:38805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adac9fd451ee40cd84acb12a728da833: Network error: Client connection negotiation failed: client connection to 127.24.141.2:38805: connect: Connection refused (error 111)
I20260620 12:15:41.764781 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:15:41.764285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:41581 (local address 127.0.0.1:39242)
0620 12:15:41.764569 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 12:15:41.764638 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.3:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 12:15:41.765272 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 3905
I20260620 12:15:41.776818 25140 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 12:15:42.123319 25140 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 12:15:42.123543 25140 mini_ranger.cc:66] Stopping Ranger...
I20260620 12:15:42.462261 25140 mini_ranger.cc:68] Stopped Ranger
2026-06-20 12:15:42.462 UTC [3488] LOG: received smart shutdown request
2026-06-20 12:15:42.466 UTC [3488] LOG: background worker "logical replication launcher" (PID 3498) exited with exit code 1
2026-06-20 12:15:42.466 UTC [3493] LOG: shutting down
2026-06-20 12:15:42.653 UTC [3488] LOG: database system is shut down
2026-06-20T12:15:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (56728 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:15:42 dist-test-slave-rh5v krb5kdc[4477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:15:42 dist-test-slave-rh5v krb5kdc[4477](info): set up 2 sockets
Jun 20 12:15:42 dist-test-slave-rh5v krb5kdc[4477](info): commencing operation
krb5kdc: starting...
W20260620 12:15:43.753527 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:15:43 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:15:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/rangeradmin_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/rangeradmin_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/rangerlookup_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/rangerlookup_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
I20260620 12:15:43.947820 25140 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task23Ce8q/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/postgres -l logfile start
2026-06-20 12:15:46.661 UTC [4507] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 12:15:46.661 UTC [4507] LOG: listening on IPv4 address "127.24.141.20", port 35767
2026-06-20 12:15:46.665 UTC [4507] LOG: listening on Unix socket "/tmp/.s.PGSQL.35767"
2026-06-20 12:15:46.689 UTC [4510] LOG: database system was shut down at 2026-06-20 12:15:45 UTC
2026-06-20 12:15:46.694 UTC [4507] LOG: database system is ready to accept connections
I20260620 12:15:47.565255 25140 mini_postgres.cc:96] Postgres bound to 35767
2026-06-20 12:15:47.572 UTC [4518] FATAL: database "slave" does not exist
127.24.141.20:35767 - accepting connections
I20260620 12:15:47.573678 25140 mini_ranger.cc:161] Starting Ranger...
I20260620 12:15:47.592242 25140 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 12:15:48.108565 25140 mini_ranger.cc:87] Created ranger Postgres database
I20260620 12:15:48.108680 25140 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-admin
2026-06-20 12:15:48,409 [I] DB FLAVOR :POSTGRES
2026-06-20 12:15:48,409 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:15:48,410 [I] Checking connection..
2026-06-20 12:15:48,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 12:15:48,705 [I] Checking connection passed.
2026-06-20 12:15:48,706 [I] --------- Verifying version history table ---------
2026-06-20 12:15:48,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:15:49,001 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:15:49,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:15:49,269 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:15:49,269 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 12:15:49,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 12:15:49.535 UTC [4582] WARNING: there is no transaction in progress
2026-06-20 12:15:49,542 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 12:15:49,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:15:49,841 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 12:15:49,841 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:15:49,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 12:15:50,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 12:15:50,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rh5v.c.gcp-upstream.internal','N') ;"
2026-06-20 12:15:50,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:15:50,962 [I] Table x_portal_user does not exist in database ranger
2026-06-20 12:15:50,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:15:51,244 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 12:15:51,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:15:51,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:15:51,820 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 12:15:51,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 12:15:52.971 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:52.990 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.011 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.038 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.060 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.084 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.151 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.159 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.177 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.190 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.203 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.220 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.231 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.240 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.249 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:53.260 UTC [4690] WARNING: there is no transaction in progress
2026-06-20 12:15:54,059 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 12:15:54,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 12:15:54,336 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 12:15:54,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:15:54,622 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 12:15:54,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:15:54,909 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 12:15:54,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:15:55,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:15:55,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rh5v.c.gcp-upstream.internal';"
2026-06-20 12:15:55,791 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 12:15:55,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 12:15:56,095 [I] DB_PATCHES have already been applied
I20260620 12:15:56.102058 25140 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-admin:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task23Ce8q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 12:15:56.102164 25140 mini_ranger.cc:193] Using host: 127.24.141.20
I20260620 12:15:56.106976 25140 mini_ranger.cc:227] Ranger admin URL: http://127.24.141.20:58441
Jun 20, 2026 12:15:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:15:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:15:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:15:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:15:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:15:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:15:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 12:15:58.219096 25140 mini_ranger.cc:160] Time spent starting Ranger: real 10.645s user 0.000s sys 0.017s
Jun 20, 2026 12:16:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:16:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:16:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:16:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:16:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 12:16:25.420593 25140 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.141.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.141.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/rangerkms_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.141.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/rangerkms_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.141.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.141.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.141.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/HTTP_127.24.141.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 12:16:25 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957785, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 12:16:25.527170 25140 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 12:16:25.541491 25140 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 12:16:26.323170 25140 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 12:16:26.323308 25140 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms
I20260620 12:16:26.323329 25140 mini_ranger_kms.cc:227] Using postgres at 127.24.141.20:35767
2026-06-20 12:16:26,549 [I] DB FLAVOR :POSTGRES
2026-06-20 12:16:26,549 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:16:26,549 [I] Checking connection
2026-06-20 12:16:26,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 12:16:26,844 [I] connection success
2026-06-20 12:16:26,844 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 12:16:26,845 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 12:16:26,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 12:16:27,196 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 12:16:27,197 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:16:27,197 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 12:16:27,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.141.20:35767/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 12:16:27,524 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 12:16:28.135494 25140 mini_ranger_kms.cc:310] Created kms service
I20260620 12:16:28.399078 25140 mini_ranger_kms.cc:326] Created kudu user
I20260620 12:16:28.449311 25140 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 12:16:28 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957785, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:28.981429 25140 mini_ranger_kms.cc:384] Added ranger policy
I20260620 12:16:28.981659 25140 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms:/tmp/dist-test-task23Ce8q/build/debug/bin/postgresql.jar:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task23Ce8q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 12:16:28.981680 25140 mini_ranger_kms.cc:240] Using host: 127.24.141.20
I20260620 12:16:28.986536 25140 mini_ranger_kms.cc:276] Ranger KMS PID: 4872
I20260620 12:16:28.986647 25140 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.141.20:47815
Jun 20, 2026 12:16:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:16:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:16:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:16:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:16:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:16:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 12:16:31.067158 25140 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.540s user 0.020s sys 0.193s
I20260620 12:16:31.067329 25140 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 12:16:31.067417 25140 mini_ranger_kms.cc:401] 127.24.141.20:47815/kms/v1/keys
Jun 20 12:16:31 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781957791, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.141.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 12:16:31 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957791, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.141.20@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
Jun 20 12:16:32 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957791, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.141.20@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
Jun 20 12:16:35 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:16:35.104570 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:32901
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:32901
--ranger_config_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:35.237432 4922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:35.237812 4922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:35.237862 4922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:35.242792 4922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:35.243068 4922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:35.243144 4922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:35.243181 4922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:35.243211 4922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:35.251435 4922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.141.20:47815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:32901
--ranger_config_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:32901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:35.252828 4922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:35.254163 4922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:16:35.267542 4922 server_base.cc:1034] running on GCE node
W20260620 12:16:35.267570 4927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:35.267834 4930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:35.267591 4928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:35.268648 4922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:35.270187 4922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:35.271442 4922 hybrid_clock.cc:648] HybridClock initialized: now 1781957795271435 us; error 39 us; skew 500 ppm
Jun 20 12:16:35 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:35.275534 4922 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:16:35.277585 4922 webserver.cc:460] Webserver started at http://127.24.141.62:45573/ using document root <none> and password file <none>
I20260620 12:16:35.279196 4922 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:35.279320 4922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:35.279597 4922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:16:35 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:35.346380 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "df46d2f62f274b03a823e359711e94f5"
format_stamp: "Formatted at 2026-06-20 12:16:35 on dist-test-slave-rh5v"
server_key: "a7edcc592a92d192b06259a712864735"
server_key_iv: "558efa4f3042ad69c07876229cda5293"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:35.347133 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "df46d2f62f274b03a823e359711e94f5"
format_stamp: "Formatted at 2026-06-20 12:16:35 on dist-test-slave-rh5v"
server_key: "a7edcc592a92d192b06259a712864735"
server_key_iv: "558efa4f3042ad69c07876229cda5293"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:35.351958 4922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:16:35.372081 4937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:35.374465 4922 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 12:16:35.374640 4922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "df46d2f62f274b03a823e359711e94f5"
format_stamp: "Formatted at 2026-06-20 12:16:35 on dist-test-slave-rh5v"
server_key: "a7edcc592a92d192b06259a712864735"
server_key_iv: "558efa4f3042ad69c07876229cda5293"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:35.374830 4922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:35.405566 4922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:35.408946 4922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:35.409216 4922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:35.418107 4922 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:32901
I20260620 12:16:35.418118 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:32901 every 8 connection(s)
I20260620 12:16:35.419284 4922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:16:35.422426 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:35.425400 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 4922
I20260620 12:16:35.425588 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:16:35.425966 25140 external_mini_cluster.cc:1428] Setting key 8dc7e67300b8fbb89a48738d38ac6d1f
I20260620 12:16:35.433777 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Bootstrap starting.
I20260620 12:16:35.437202 4990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:35.438228 4990 log.cc:824] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:35.441232 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: No bootstrap required, opened a new log
Jun 20 12:16:35 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:35.445986 4990 raft_consensus.cc:348] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:35.446322 4990 raft_consensus.cc:374] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:35.446414 4990 raft_consensus.cc:729] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df46d2f62f274b03a823e359711e94f5, State: Initialized, Role: FOLLOWER
I20260620 12:16:35.447024 4990 consensus_queue.cc:260] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:35.447208 4990 raft_consensus.cc:388] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:16:35.447301 4990 raft_consensus.cc:482] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:16:35.447413 4990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:35.448633 4990 raft_consensus.cc:504] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:35.449198 4990 leader_election.cc:302] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df46d2f62f274b03a823e359711e94f5; no voters:
I20260620 12:16:35.449633 4990 leader_election.cc:288] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:16:35.449807 4995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:16:35.450035 4995 raft_consensus.cc:686] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 LEADER]: Becoming Leader. State: Replica: df46d2f62f274b03a823e359711e94f5, State: Running, Role: LEADER
I20260620 12:16:35.450448 4995 consensus_queue.cc:237] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:35.451251 4990 sys_catalog.cc:564] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:35.451996 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:35.428661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.24.141.62:32901)
0620 12:16:35.429438 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:35.429466 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:35.429550 (+ 84us) server_negotiation.cc:407] Connection header received
0620 12:16:35.430883 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:35.430924 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:35.431708 (+ 784us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:35.432313 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:35.433494 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:35.435339 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:35.437164 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:35.438406 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:35.438713 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:35.443259 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:35.443271 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:35.443383 (+ 112us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:35.443440 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:35.445862 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:35.446759 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:35.446766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:35.446774 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:35.446960 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:35.447410 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:35.447417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:35.447419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:35.448441 (+ 1022us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:35.449837 (+ 1396us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:35.450067 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 12:16:35.450559 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":231,"threads_started":1}
I20260620 12:16:35.452101 4995 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df46d2f62f274b03a823e359711e94f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } } }
I20260620 12:16:35.452801 4995 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:35.453171 4995 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df46d2f62f274b03a823e359711e94f5. Latest consensus state: current_term: 1 leader_uuid: "df46d2f62f274b03a823e359711e94f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } } }
I20260620 12:16:35.453291 4995 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:35.453892 4990 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:16:35.461642 5004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:16:35.467406 5004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:16:35.478837 5004 catalog_manager.cc:1347] Generated new cluster ID: 2be6e8194b4f445c86df2aa7e2a6cc6c
I20260620 12:16:35.478960 5004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:16:35.494609 5004 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:16:35.496577 5004 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:16:35.516964 5004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Generated new TSK 0
I20260620 12:16:35.518563 5004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:16:36.385560 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:32901
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:36.575119 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:36.575752 5035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:36.575947 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:36.583315 5035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:36.583457 5035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:36.583585 5035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:16:36.594043 5035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.141.20:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:32901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:36.596284 5035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:36.598325 5035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:36.615458 5042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:36.615598 5045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:36.615471 5043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:36.615967 5035 server_base.cc:1034] running on GCE node
I20260620 12:16:36.616886 5035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:36.618228 5035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:36.619510 5035 hybrid_clock.cc:648] HybridClock initialized: now 1781957796619458 us; error 72 us; skew 500 ppm
Jun 20 12:16:36 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:36.624491 5035 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:16:36.626094 5035 webserver.cc:460] Webserver started at http://127.24.141.1:44095/ using document root <none> and password file <none>
I20260620 12:16:36.626966 5035 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:36.627045 5035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:36.627285 5035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:16:36 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:36.670588 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "825f7a9c6c844b2696c499e85a81707c"
format_stamp: "Formatted at 2026-06-20 12:16:36 on dist-test-slave-rh5v"
server_key: "715d4b77b58461d131ed1332db4d0d91"
server_key_iv: "b630bf693d0da04345861dee7059a0b9"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:36.672405 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "825f7a9c6c844b2696c499e85a81707c"
format_stamp: "Formatted at 2026-06-20 12:16:36 on dist-test-slave-rh5v"
server_key: "715d4b77b58461d131ed1332db4d0d91"
server_key_iv: "b630bf693d0da04345861dee7059a0b9"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:36.680032 5035 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 12:16:36.710229 5054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:36.712745 5035 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260620 12:16:36.712960 5035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "825f7a9c6c844b2696c499e85a81707c"
format_stamp: "Formatted at 2026-06-20 12:16:36 on dist-test-slave-rh5v"
server_key: "715d4b77b58461d131ed1332db4d0d91"
server_key_iv: "b630bf693d0da04345861dee7059a0b9"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:36.713187 5035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:36.739329 5035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:36.744946 5035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:36.745359 5035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:36.746464 5035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:36.748255 5035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:36.748413 5035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 12:16:36.748584 5035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:36.748782 5035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:36.768997 5035 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:44737
I20260620 12:16:36.772235 5035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:16:36.774748 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 5035
I20260620 12:16:36.774909 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:16:36.775305 25140 external_mini_cluster.cc:1428] Setting key 5b77615d9fae4bfb1bc73918f16727bb
I20260620 12:16:36.773484 5167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:44737 every 8 connection(s)
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
Jun 20 12:16:36 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781957796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 12:16:36 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:36.837641 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:36.786357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:35557 (local address 127.24.141.62:32901)
0620 12:16:36.796220 (+ 9863us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:36.796229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:36.796251 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:16:36.796302 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:36.796305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:36.796362 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:36.796471 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:36.805814 (+ 9343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:36.806779 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:36.813331 (+ 6552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:36.813799 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:36.813926 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:36.825764 (+ 11838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:36.825770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:36.825773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:36.825823 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:36.828172 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:36.829406 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:36.829412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:36.829414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:36.829560 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:36.834893 (+ 5333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:36.834904 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:36.834907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:36.835425 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:36.835609 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:36.836918 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0620 12:16:36.837355 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9733,"thread_start_us":118,"threads_started":1}
I20260620 12:16:36.840883 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:36.784713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:32901 (local address 127.24.141.1:35557)
0620 12:16:36.793733 (+ 9020us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:36.793868 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:36.795577 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:36.798960 (+ 3383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:36.798978 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:36.799481 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:36.800860 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:36.800901 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:36.807164 (+ 6263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:36.807170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:36.810827 (+ 3657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:36.810869 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:36.817248 (+ 6379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:36.817255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:36.817678 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:36.822581 (+ 4903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:36.822624 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:36.825333 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:36.828492 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:36.828503 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:36.828538 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:36.829226 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:36.829709 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:36.829713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:36.829715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:36.830024 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:36.835777 (+ 5753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:36.835791 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:36.836449 (+ 658us) client_negotiation.cc:769] Sending connection context
0620 12:16:36.838390 (+ 1941us) client_negotiation.cc:240] Negotiation successful
0620 12:16:36.839150 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8504,"thread_start_us":248,"threads_started":1}
I20260620 12:16:36.843680 5168 heartbeater.cc:347] Connected to a master server at 127.24.141.62:32901
I20260620 12:16:36.844201 5168 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:36.845350 5168 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:36.848428 4953 ts_manager.cc:194] Registered new tserver with Master: 825f7a9c6c844b2696c499e85a81707c (127.24.141.1:44737)
I20260620 12:16:36.851188 4953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:35557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:16:36.861552 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:32901
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:37.091277 5174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:37.092298 5174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:37.092444 5174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:37.101002 5174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:37.101260 5174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:37.101424 5174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:16:37.112078 5174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.141.20:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:32901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:37.115077 5174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:37.117740 5174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:37.135918 5180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:37.136375 5174 server_base.cc:1034] running on GCE node
W20260620 12:16:37.136512 5183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:37.135846 5181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:37.142046 5174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:37.143093 5174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:37.146404 5174 hybrid_clock.cc:648] HybridClock initialized: now 1781957797146301 us; error 112 us; skew 500 ppm
Jun 20 12:16:37 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:37.150799 5174 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:16:37.152238 5174 webserver.cc:460] Webserver started at http://127.24.141.2:44465/ using document root <none> and password file <none>
I20260620 12:16:37.153175 5174 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:37.153354 5174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:37.153705 5174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:16:37 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:37.177320 5174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "3b713ec6ca3443a48454881fbff6feff"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "b9ae3ee8137c2d4069d6c0a757522805"
server_key_iv: "1bc8b2b8472741e63cfd1ae5674f26fb"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:37.181025 5174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "3b713ec6ca3443a48454881fbff6feff"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "b9ae3ee8137c2d4069d6c0a757522805"
server_key_iv: "1bc8b2b8472741e63cfd1ae5674f26fb"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:37.187892 5174 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260620 12:16:37.208804 5190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:37.212014 5174 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:16:37.212299 5174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "3b713ec6ca3443a48454881fbff6feff"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "b9ae3ee8137c2d4069d6c0a757522805"
server_key_iv: "1bc8b2b8472741e63cfd1ae5674f26fb"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:37.212515 5174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:37.247722 5174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:37.256697 5174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:37.258193 5174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:37.259176 5174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:37.260797 5174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:37.261129 5174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 12:16:37.261269 5174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:37.261338 5174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:37.274009 5174 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:35087
I20260620 12:16:37.275532 5174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:16:37.280105 5303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:35087 every 8 connection(s)
I20260620 12:16:37.286104 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 5174
I20260620 12:16:37.286309 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:16:37.286700 25140 external_mini_cluster.cc:1428] Setting key 938414c23956076a43fcea8d7d78022f
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Jun 20 12:16:37 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:16:37.313690 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:37.286667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:49329 (local address 127.24.141.62:32901)
0620 12:16:37.286857 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:37.286863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:37.287726 (+ 863us) server_negotiation.cc:407] Connection header received
0620 12:16:37.289443 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:37.289447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:37.289530 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:37.289591 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:37.292213 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.293173 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.295732 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.296145 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.296262 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:37.308276 (+ 12014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:37.308282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:37.308287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:37.308370 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:37.310442 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:37.311368 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:37.311372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:37.311374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:37.311482 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:37.311984 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:37.311988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:37.311990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:37.312257 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:37.312344 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:37.313227 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 12:16:37.313456 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:16:37.316190 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:37.284543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:32901 (local address 127.24.141.2:49329)
0620 12:16:37.287475 (+ 2932us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:37.287593 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:37.289043 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:37.290241 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:37.290255 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:37.290818 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:37.291903 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:37.291937 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.293430 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.293434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:37.295458 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:37.295483 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.296616 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.296622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:37.296935 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:37.302632 (+ 5697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:37.302664 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:37.304867 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:37.310695 (+ 5828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:37.310704 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:37.310730 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:37.311202 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:37.311620 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:37.311623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:37.311625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:37.311868 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:37.312438 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:37.312446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:37.312862 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 12:16:37.314251 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0620 12:16:37.314750 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2580,"thread_start_us":263,"threads_started":1}
I20260620 12:16:37.318346 5304 heartbeater.cc:347] Connected to a master server at 127.24.141.62:32901
I20260620 12:16:37.318670 5304 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:37.319412 5304 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:37.320925 4953 ts_manager.cc:194] Registered new tserver with Master: 3b713ec6ca3443a48454881fbff6feff (127.24.141.2:35087)
I20260620 12:16:37.321929 4953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:49329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:16:37.348212 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:32901
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:16:37 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
W20260620 12:16:37.569486 5309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:37.570192 5309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:37.570317 5309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:37.578169 5309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:37.578441 5309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:37.578610 5309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:16:37.589522 5309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.141.20:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:32901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:37.591900 5309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:37.594002 5309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:16:37.613706 5309 server_base.cc:1034] running on GCE node
W20260620 12:16:37.613874 5314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:37.613687 5315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:37.615063 5317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:37.615907 5309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:37.617023 5309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:37.620343 5309 hybrid_clock.cc:648] HybridClock initialized: now 1781957797620220 us; error 119 us; skew 500 ppm
Jun 20 12:16:37 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:37.626722 5309 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:16:37.628296 5309 webserver.cc:460] Webserver started at http://127.24.141.3:41875/ using document root <none> and password file <none>
I20260620 12:16:37.629878 5309 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:37.630163 5309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:37.630483 5309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:16:37 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:37.679783 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "28f05c39597e4144b2965a5f8fa8406e"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "99fc16227b0c3f0638899b4ea249b623"
server_key_iv: "3f956ff9185d3fc368f3224c32091c83"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:37.682047 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "28f05c39597e4144b2965a5f8fa8406e"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "99fc16227b0c3f0638899b4ea249b623"
server_key_iv: "3f956ff9185d3fc368f3224c32091c83"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:37.715703 5309 fs_manager.cc:691] Time spent creating directory manager: real 0.031s user 0.005s sys 0.003s
I20260620 12:16:37.742585 5324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:37.745261 5309 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260620 12:16:37.745540 5309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "28f05c39597e4144b2965a5f8fa8406e"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "99fc16227b0c3f0638899b4ea249b623"
server_key_iv: "3f956ff9185d3fc368f3224c32091c83"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:37.745754 5309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:37.771821 5309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:37.777863 5309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:37.778270 5309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:37.779330 5309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:37.780970 5309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:37.781160 5309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 12:16:37.781286 5309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:37.781349 5309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:37.798501 5309 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:43101
I20260620 12:16:37.800410 5309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:16:37.802589 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 5309
I20260620 12:16:37.802742 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:16:37.803157 25140 external_mini_cluster.cc:1428] Setting key b3d63c085126152c12a3b16488639c09
I20260620 12:16:37.802160 5437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:43101 every 8 connection(s)
Jun 20 12:16:37 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:37.835939 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:37.808211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:49199 (local address 127.24.141.62:32901)
0620 12:16:37.808383 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:37.808388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:37.809171 (+ 783us) server_negotiation.cc:407] Connection header received
0620 12:16:37.810828 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:37.810833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:37.810917 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:37.810977 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:37.814662 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.815542 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.819354 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.819743 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.819834 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:37.828100 (+ 8266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:37.828107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:37.828110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:37.828170 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:37.830457 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:37.831382 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:37.831387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:37.831389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:37.831595 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:37.833898 (+ 2303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:37.833903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:37.833905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:37.834251 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:37.834357 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:37.835379 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0620 12:16:37.835716 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:16:37.839465 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:37.806638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:32901 (local address 127.24.141.3:49199)
0620 12:16:37.808880 (+ 2242us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:37.808965 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:37.810463 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:37.811853 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:37.811868 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:37.812361 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:37.814351 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:37.814387 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.815761 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.815765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:37.819167 (+ 3402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:37.819192 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:37.820210 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:37.820215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:37.820551 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:37.825456 (+ 4905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:37.825492 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:37.827650 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:37.830695 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:37.830705 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:37.830731 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:37.831195 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:37.833235 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:37.833240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:37.833243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:37.833714 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:37.834457 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:37.834469 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:37.834997 (+ 528us) client_negotiation.cc:769] Sending connection context
0620 12:16:37.836649 (+ 1652us) client_negotiation.cc:240] Negotiation successful
0620 12:16:37.837323 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1926,"thread_start_us":230,"threads_started":1}
I20260620 12:16:37.843067 5438 heartbeater.cc:347] Connected to a master server at 127.24.141.62:32901
I20260620 12:16:37.843508 5438 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:37.844460 5438 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:37.848009 4953 ts_manager.cc:194] Registered new tserver with Master: 28f05c39597e4144b2965a5f8fa8406e (127.24.141.3:43101)
I20260620 12:16:37.849045 4953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:49199
I20260620 12:16:37.857262 5168 heartbeater.cc:503] Master 127.24.141.62:32901 was elected leader, sending a full tablet report...
I20260620 12:16:37.860706 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:16:37.861949 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 5035
W20260620 12:16:37.872202 4939 connection.cc:535] server connection from 127.24.141.1:35557 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:16:37.873790 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 5174
I20260620 12:16:37.884531 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 5309
I20260620 12:16:37.893908 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 4922
I20260620 12:16:37.902805 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:32901
--webserver_interface=127.24.141.62
--webserver_port=45573
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:32901
--ranger_config_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:38.073331 5441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:38.073791 5441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:38.073841 5441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:38.078922 5441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:38.079057 5441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:38.079078 5441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:38.079094 5441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:38.079113 5441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:38.086405 5441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:32901
--ranger_config_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:32901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=45573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:38.088033 5441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:38.089533 5441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:38.101199 5449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:38.101207 5446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:38.101199 5447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:38.102626 5441 server_base.cc:1034] running on GCE node
I20260620 12:16:38.103582 5441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:38.105407 5441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:38.106643 5441 hybrid_clock.cc:648] HybridClock initialized: now 1781957798106609 us; error 62 us; skew 500 ppm
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:38.110747 5441 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:16:38.112123 5441 webserver.cc:460] Webserver started at http://127.24.141.62:45573/ using document root <none> and password file <none>
I20260620 12:16:38.112844 5441 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:38.112937 5441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:38.201020 5441 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.003s
I20260620 12:16:38.205812 5457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:38.207958 5441 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 12:16:38.208173 5441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "df46d2f62f274b03a823e359711e94f5"
format_stamp: "Formatted at 2026-06-20 12:16:35 on dist-test-slave-rh5v"
server_key: "a7edcc592a92d192b06259a712864735"
server_key_iv: "558efa4f3042ad69c07876229cda5293"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:38.208869 5441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:38.244047 5441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:38.248759 5441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:38.249050 5441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:38.260619 5441 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:32901
I20260620 12:16:38.260747 5509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:32901 every 8 connection(s)
I20260620 12:16:38.262475 5441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:16:38.267809 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 5441
I20260620 12:16:38.268383 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:44737
--local_ip_for_outbound_sockets=127.24.141.1
--tserver_master_addrs=127.24.141.62:32901
--webserver_port=44095
--webserver_interface=127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:16:38.269029 5510 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:16:38.271311 5510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Bootstrap starting.
I20260620 12:16:38.291893 5510 log.cc:824] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:38.301935 5510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:16:38.302481 5510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Bootstrap complete.
I20260620 12:16:38.307778 5510 raft_consensus.cc:348] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:38.308341 5510 raft_consensus.cc:729] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df46d2f62f274b03a823e359711e94f5, State: Initialized, Role: FOLLOWER
I20260620 12:16:38.308952 5510 consensus_queue.cc:260] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:38.309136 5510 raft_consensus.cc:388] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:16:38.309226 5510 raft_consensus.cc:482] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:16:38.309343 5510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:16:38.311895 5510 raft_consensus.cc:504] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:38.312386 5510 leader_election.cc:302] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df46d2f62f274b03a823e359711e94f5; no voters:
I20260620 12:16:38.312773 5510 leader_election.cc:288] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:16:38.312898 5514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:16:38.313133 5514 raft_consensus.cc:686] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [term 2 LEADER]: Becoming Leader. State: Replica: df46d2f62f274b03a823e359711e94f5, State: Running, Role: LEADER
I20260620 12:16:38.313628 5514 consensus_queue.cc:237] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } }
I20260620 12:16:38.314603 5510 sys_catalog.cc:564] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:38.315488 5516 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df46d2f62f274b03a823e359711e94f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } } }
I20260620 12:16:38.315718 5516 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:38.316743 5515 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df46d2f62f274b03a823e359711e94f5. Latest consensus state: current_term: 2 leader_uuid: "df46d2f62f274b03a823e359711e94f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df46d2f62f274b03a823e359711e94f5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 32901 } } }
I20260620 12:16:38.317001 5515 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:38.317241 5523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:16:38.317427 5510 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:16:38.325845 5523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:16:38.326718 5523 catalog_manager.cc:1259] Loaded cluster ID: 2be6e8194b4f445c86df2aa7e2a6cc6c
I20260620 12:16:38.326802 5523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:16:38.329780 5523 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:16:38.331559 5523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Loaded TSK: 0
I20260620 12:16:38.334462 5523 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:16:38.428581 5512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:38.428932 5512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:38.429000 5512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:38.434749 5512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:38.434859 5512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:38.434950 5512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:16:38.443969 5512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:44737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=44095
--tserver_master_addrs=127.24.141.62:32901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:38.445809 5512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:38.447553 5512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:38.460650 5548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:38.461140 5512 server_base.cc:1034] running on GCE node
W20260620 12:16:38.460731 5550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:38.461377 5547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:38.461875 5512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:38.462666 5512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:38.463877 5512 hybrid_clock.cc:648] HybridClock initialized: now 1781957798463856 us; error 36 us; skew 500 ppm
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:38.467156 5512 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:16:38.468461 5512 webserver.cc:460] Webserver started at http://127.24.141.1:44095/ using document root <none> and password file <none>
I20260620 12:16:38.469318 5512 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:38.469432 5512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:38.499022 5512 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 12:16:38.503760 5558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:38.506595 5512 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 12:16:38.506875 5512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "825f7a9c6c844b2696c499e85a81707c"
format_stamp: "Formatted at 2026-06-20 12:16:36 on dist-test-slave-rh5v"
server_key: "715d4b77b58461d131ed1332db4d0d91"
server_key_iv: "b630bf693d0da04345861dee7059a0b9"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:38.507994 5512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:38.540470 5512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:38.545780 5512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:38.546584 5512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:38.548053 5512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:38.550326 5512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:38.550455 5512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:38.550711 5512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:38.550760 5512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:38.571007 5512 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:44737
I20260620 12:16:38.575251 5671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:44737 every 8 connection(s)
I20260620 12:16:38.575871 5512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:16:38.582888 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 5512
I20260620 12:16:38.583482 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:35087
--local_ip_for_outbound_sockets=127.24.141.2
--tserver_master_addrs=127.24.141.62:32901
--webserver_port=44465
--webserver_interface=127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:38.608946 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:38.580138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:32901 (local address 127.24.141.1:50249)
0620 12:16:38.580982 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:38.581051 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:38.582304 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:38.584958 (+ 2654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:38.584972 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:38.585799 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:38.586605 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:38.586631 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.588455 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.588460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:38.590012 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:38.590036 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.591859 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.591866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:38.592103 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:38.592844 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:38.592872 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:38.597745 (+ 4873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:38.602981 (+ 5236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:38.602993 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:38.603021 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:38.603754 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:38.604599 (+ 845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:38.604604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:38.604607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:38.604930 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:38.606266 (+ 1336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:38.606279 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:38.606820 (+ 541us) client_negotiation.cc:769] Sending connection context
0620 12:16:38.607157 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 12:16:38.607620 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":232,"threads_started":1}
I20260620 12:16:38.610657 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:38.579671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:50249 (local address 127.24.141.62:32901)
0620 12:16:38.580243 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:38.580264 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:38.581274 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 12:16:38.583306 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:38.583328 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:38.584084 (+ 756us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:38.584720 (+ 636us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:38.586938 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.588188 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.590249 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.591269 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.591512 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:38.598518 (+ 7006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:38.598535 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:38.598566 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:38.598631 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:38.602633 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:38.604067 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:38.604080 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:38.604091 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:38.604404 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:38.605124 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:38.605129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:38.605131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:38.605982 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:38.606207 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:38.607462 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0620 12:16:38.608003 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":162,"threads_started":1}
I20260620 12:16:38.615464 5672 heartbeater.cc:347] Connected to a master server at 127.24.141.62:32901
I20260620 12:16:38.615854 5672 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:38.616669 5672 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
W20260620 12:16:38.621505 5672 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:32901 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:16:38.621795 5672 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:38.622164 5672 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:38.623917 5672 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:38.624254 5672 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
W20260620 12:16:38.625681 5672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:16:38.785977 5676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:38.786445 5676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:38.786527 5676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:38.791707 5676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:38.791808 5676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:38.791915 5676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:16:38.799868 5676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:35087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=44465
--tserver_master_addrs=127.24.141.62:32901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:38.801753 5676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:38.804179 5676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:38.820104 5684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:38.820441 5681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:38.820627 5682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:38.820932 5676 server_base.cc:1034] running on GCE node
I20260620 12:16:38.821620 5676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:38.822667 5676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:38.826149 5676 hybrid_clock.cc:648] HybridClock initialized: now 1781957798826089 us; error 93 us; skew 500 ppm
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:38.831478 5676 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:16:38.833225 5676 webserver.cc:460] Webserver started at http://127.24.141.2:44465/ using document root <none> and password file <none>
I20260620 12:16:38.834115 5676 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:38.834254 5676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:38.866690 5676 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 12:16:38.870672 5692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:38.873337 5676 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260620 12:16:38.873662 5676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "3b713ec6ca3443a48454881fbff6feff"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "b9ae3ee8137c2d4069d6c0a757522805"
server_key_iv: "1bc8b2b8472741e63cfd1ae5674f26fb"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:38.874886 5676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:38.903993 5676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:38.911697 5676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:38.912163 5676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:38.913625 5676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:38.915258 5676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:38.915365 5676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:38.915443 5676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:38.915470 5676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:38.939323 5676 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:35087
I20260620 12:16:38.939458 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:35087 every 8 connection(s)
I20260620 12:16:38.941619 5676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:16:38.948292 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 5676
I20260620 12:16:38.948897 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.141.20:47815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:43101
--local_ip_for_outbound_sockets=127.24.141.3
--tserver_master_addrs=127.24.141.62:32901
--webserver_port=41875
--webserver_interface=127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:16:38 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:38.966212 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:38.944974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:34303 (local address 127.24.141.62:32901)
0620 12:16:38.945175 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:38.945179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:38.946896 (+ 1717us) server_negotiation.cc:407] Connection header received
0620 12:16:38.948894 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:38.948900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:38.949007 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:38.949112 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:38.951921 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.952862 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.954854 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.955220 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.955345 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:38.959871 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:38.959878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:38.959881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:38.959927 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:38.962247 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:38.963527 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:38.963533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:38.963534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:38.963694 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:38.964210 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:38.964213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:38.964215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:38.964541 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:38.964660 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:38.965706 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0620 12:16:38.965969 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:16:38.967767 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:38.945708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:32901 (local address 127.24.141.2:34303)
0620 12:16:38.946596 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:38.946704 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:38.948344 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:38.950030 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:38.950046 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:38.950599 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:38.951624 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:38.951654 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.953139 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.953143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:38.954640 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:38.954663 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:38.955503 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:38.955509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:38.955807 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:38.957195 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:38.957228 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:38.959489 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:38.962483 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:38.962494 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:38.962520 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:38.963298 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:38.963838 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:38.963842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:38.963844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:38.964082 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:38.964673 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:38.964685 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:38.965244 (+ 559us) client_negotiation.cc:769] Sending connection context
0620 12:16:38.965727 (+ 483us) client_negotiation.cc:240] Negotiation successful
0620 12:16:38.966381 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":190,"threads_started":1}
I20260620 12:16:38.969779 5806 heartbeater.cc:347] Connected to a master server at 127.24.141.62:32901
I20260620 12:16:38.970257 5806 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:38.971149 5806 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
W20260620 12:16:38.974327 5806 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:32901 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:16:38.974586 5806 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:38.974906 5806 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:38.976547 5806 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:38.976866 5806 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
W20260620 12:16:38.978163 5806 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:16:39.160570 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:39.161006 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:39.161108 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:39.167485 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:39.167635 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:39.167732 5809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:16:39.175231 5809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.141.20:47815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:43101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=41875
--tserver_master_addrs=127.24.141.62:32901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:39.176913 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:39.178423 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:39.191694 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:39.191677 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:39.191679 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:39.192298 5809 server_base.cc:1034] running on GCE node
I20260620 12:16:39.193135 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:39.194561 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:39.198082 5809 hybrid_clock.cc:648] HybridClock initialized: now 1781957799197938 us; error 148 us; skew 500 ppm
Jun 20 12:16:39 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957799, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:39.205031 5809 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:16:39.207087 5809 webserver.cc:460] Webserver started at http://127.24.141.3:41875/ using document root <none> and password file <none>
I20260620 12:16:39.208091 5809 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:39.208249 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:16:39 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957799, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:39.238835 5809 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:16:39.246196 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:39.248736 5809 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260620 12:16:39.249015 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "28f05c39597e4144b2965a5f8fa8406e"
format_stamp: "Formatted at 2026-06-20 12:16:37 on dist-test-slave-rh5v"
server_key: "99fc16227b0c3f0638899b4ea249b623"
server_key_iv: "3f956ff9185d3fc368f3224c32091c83"
server_key_version: "kuduclusterkey@0"
I20260620 12:16:39.250069 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:39.307446 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:39.311121 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:39.311379 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:39.312278 5809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:39.313791 5809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:39.313887 5809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:39.313963 5809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:39.313999 5809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:39.328286 5809 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:43101
I20260620 12:16:39.328743 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:43101 every 8 connection(s)
I20260620 12:16:39.330009 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:16:39.335230 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 5809
Jun 20 12:16:39 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957799, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:39.350813 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:39.333400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:34159 (local address 127.24.141.62:32901)
0620 12:16:39.333565 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:39.333571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:39.334257 (+ 686us) server_negotiation.cc:407] Connection header received
0620 12:16:39.335667 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:39.335672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:39.335744 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:39.335805 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:39.338101 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:39.339046 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:39.341034 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:39.341417 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:39.341536 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:39.345510 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:39.345516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:39.345519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:39.345557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:39.347821 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:39.348723 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:39.348727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:39.348729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:39.348833 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:39.349358 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:39.349362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:39.349364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:39.349671 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:39.349791 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:39.350448 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 12:16:39.350616 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:16:39.352064 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:39.333355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:32901 (local address 127.24.141.3:34159)
0620 12:16:39.334051 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:39.334114 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:39.335287 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:39.336609 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:39.336618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:39.336942 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:39.337879 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:39.337901 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:39.339241 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:39.339245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:39.340867 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:39.340885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:39.341627 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:39.341632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:39.341892 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:39.342635 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:39.342848 (+ 213us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:39.345156 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:39.348053 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:39.348064 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:39.348089 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:39.348588 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:39.348948 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:39.348951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:39.348953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:39.349232 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:39.349803 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:39.349826 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:39.350157 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 12:16:39.350358 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 12:16:39.350747 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":198,"threads_started":1}
I20260620 12:16:39.353592 5940 heartbeater.cc:347] Connected to a master server at 127.24.141.62:32901
I20260620 12:16:39.353897 5940 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:39.354631 5940 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:39.355157 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:39.336048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.24.141.62:32901)
0620 12:16:39.338409 (+ 2361us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:39.338422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:39.338823 (+ 401us) server_negotiation.cc:407] Connection header received
0620 12:16:39.338985 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:39.338989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:39.339109 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:39.339263 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:39.340465 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:39.341462 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:39.343500 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:39.343924 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:39.344079 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:39.346409 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:39.346414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:39.346417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:39.346494 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:39.348563 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:39.349288 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:39.349295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:39.349297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:39.349617 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:39.353549 (+ 3932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:39.353557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:39.353559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:39.354029 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:39.354386 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:39.354641 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 12:16:39.354903 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2086,"thread_start_us":102,"threads_started":1}
W20260620 12:16:39.356796 5940 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:32901 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:16:39.356968 5940 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:39.357232 5940 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:39.358645 5940 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:39.358886 5940 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
W20260620 12:16:39.359712 5940 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:16:39.626114 5672 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:39.626641 5672 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:39.628513 5471 ts_manager.cc:194] Registered new tserver with Master: 825f7a9c6c844b2696c499e85a81707c (127.24.141.1:44737)
I20260620 12:16:39.630190 5471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:50249
Jun 20 12:16:39 dist-test-slave-rh5v krb5kdc[4477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781957799, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:39.978542 5806 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:39.978948 5806 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:39.980396 5471 ts_manager.cc:194] Registered new tserver with Master: 3b713ec6ca3443a48454881fbff6feff (127.24.141.2:35087)
I20260620 12:16:39.981441 5471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:34303
I20260620 12:16:40.359990 5940 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:40.360369 5940 heartbeater.cc:511] Master 127.24.141.62:32901 requested a full tablet report, sending...
I20260620 12:16:40.361802 5471 ts_manager.cc:194] Registered new tserver with Master: 28f05c39597e4144b2965a5f8fa8406e (127.24.141.3:43101)
I20260620 12:16:40.362794 5471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:34159
I20260620 12:16:40.370756 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:16:40 dist-test-slave-rh5v krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957799, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for HTTP/127.24.141.20@KRBTEST.COM
I20260620 12:16:40.393594 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.373832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.24.141.62:32901)
0620 12:16:40.378421 (+ 4589us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:40.378427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:40.378862 (+ 435us) server_negotiation.cc:407] Connection header received
0620 12:16:40.379035 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:40.379040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:40.379141 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:40.379291 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:40.380838 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.381775 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.383497 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.383893 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.384040 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.386194 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:40.386200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:40.386203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:40.386248 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:40.390730 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:40.391409 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:40.391413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:40.391415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:40.391522 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:40.391850 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:40.391853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:40.391856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:40.392116 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:40.392254 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:40.393205 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 12:16:40.393391 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4430,"thread_start_us":190,"threads_started":1}
I20260620 12:16:40.400346 5471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:16:40.409613 5471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:16:40.458881 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.429901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:43101 (local address 127.0.0.1:49312)
0620 12:16:40.436968 (+ 7067us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x28126f68. stack: 00007fb5a6b09c78 00007fb5a6b0a1a9 00007fb5a66cc1d7 00000000005984a6 00000000005984f9 000000000059c488 00007fb5adcf5f6d 00007fb5ab19d23d 00007fb5ab19e68c 00007fb5ab1b40d3 00007fb5ab1b754b 000000000061d837 00007fb5a6b2a3d6 00007fb5a6b2ac82 00007fb5a6b2c3f1 000000000061d837
0620 12:16:40.437029 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:40.437047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:40.441748 (+ 4701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:40.442610 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:40.442613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:40.442633 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:40.443079 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.443089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.444755 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.444760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.446808 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.446829 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.458116 (+ 11287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.458119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.458305 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.458352 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 12:16:40.458504 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 12:16:40.458674 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4345,"spinlock_wait_cycles":4775680,"thread_start_us":139,"threads_started":1}
I20260620 12:16:40.459488 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.430022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.24.141.3:43101)
0620 12:16:40.434111 (+ 4089us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:40.434122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:40.437214 (+ 3092us) server_negotiation.cc:407] Connection header received
0620 12:16:40.441948 (+ 4734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:40.441958 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:40.442239 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:40.442411 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:40.443229 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.444399 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.456414 (+ 12015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.457524 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.457646 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.457960 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:40.459192 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0620 12:16:40.459335 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3912,"thread_start_us":104,"threads_started":1}
I20260620 12:16:40.463330 5874 tablet_service.cc:1467] Processing CreateTablet for tablet 81b4446f913c42e08348c410a231bddc (DEFAULT_TABLE table=test-table [id=89c75921b0274577b27fff8c2975fd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:40.464852 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b4446f913c42e08348c410a231bddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:40.472213 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.427905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:35087 (local address 127.0.0.1:35086)
0620 12:16:40.435750 (+ 7845us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:40.435841 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:40.436046 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:40.441263 (+ 5217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:40.441273 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:40.441327 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:40.441547 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.441557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.453260 (+ 11703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.453266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.454550 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.454577 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.471598 (+ 17021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.471605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.471752 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.471793 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 12:16:40.471906 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 12:16:40.472008 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4982,"thread_start_us":166,"threads_started":1}
I20260620 12:16:40.472998 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.428025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.24.141.2:35087)
0620 12:16:40.437670 (+ 9645us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:40.437679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:40.437704 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:16:40.437810 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:40.437817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:40.438096 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:40.438292 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:40.447059 (+ 8767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.448271 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.455348 (+ 7077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.470441 (+ 15093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.470642 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.470947 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:40.472669 (+ 1722us) server_negotiation.cc:299] Negotiation successful
0620 12:16:40.472844 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9485,"thread_start_us":125,"threads_started":1}
I20260620 12:16:40.478016 5738 tablet_service.cc:1467] Processing CreateTablet for tablet 81b4446f913c42e08348c410a231bddc (DEFAULT_TABLE table=test-table [id=89c75921b0274577b27fff8c2975fd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:40.479596 5738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b4446f913c42e08348c410a231bddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:40.501200 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.430068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:44737 (local address 127.0.0.1:49240)
0620 12:16:40.454964 (+ 24896us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:40.454980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:40.455196 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:40.484348 (+ 29152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:40.484354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:40.484401 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:40.484541 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.484551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.486473 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.486478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.487684 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.487700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.500641 (+ 12941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.500646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.500769 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.500801 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 12:16:40.500887 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:16:40.500981 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24787,"thread_start_us":147,"threads_started":1}
I20260620 12:16:40.502004 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.430280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.24.141.1:44737)
0620 12:16:40.471308 (+ 41028us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:40.471319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:40.471441 (+ 122us) server_negotiation.cc:407] Connection header received
0620 12:16:40.471576 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:40.471584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:40.471940 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:40.476508 (+ 4568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:40.484782 (+ 8274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.486268 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.498917 (+ 12649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.500159 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.500345 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.500532 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:40.501635 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 12:16:40.501787 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40825,"thread_start_us":83,"threads_started":1}
I20260620 12:16:40.503646 5973 tablet_bootstrap.cc:492] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Bootstrap starting.
I20260620 12:16:40.506624 5605 tablet_service.cc:1467] Processing CreateTablet for tablet 81b4446f913c42e08348c410a231bddc (DEFAULT_TABLE table=test-table [id=89c75921b0274577b27fff8c2975fd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:40.508464 5605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b4446f913c42e08348c410a231bddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:40.508631 5974 tablet_bootstrap.cc:492] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Bootstrap starting.
I20260620 12:16:40.511759 5974 tablet_bootstrap.cc:654] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:40.512849 5974 log.cc:824] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:40.514782 5973 tablet_bootstrap.cc:654] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:40.515941 5973 log.cc:824] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:40.519753 5973 tablet_bootstrap.cc:492] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: No bootstrap required, opened a new log
I20260620 12:16:40.520052 5973 ts_tablet_manager.cc:1397] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260620 12:16:40.527599 5974 tablet_bootstrap.cc:492] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: No bootstrap required, opened a new log
I20260620 12:16:40.527786 5977 tablet_bootstrap.cc:492] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c: Bootstrap starting.
I20260620 12:16:40.527896 5974 ts_tablet_manager.cc:1397] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260620 12:16:40.524747 5973 raft_consensus.cc:348] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.533399 5973 raft_consensus.cc:374] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:40.533545 5973 raft_consensus.cc:729] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f05c39597e4144b2965a5f8fa8406e, State: Initialized, Role: FOLLOWER
I20260620 12:16:40.533753 5977 tablet_bootstrap.cc:654] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:40.534951 5974 raft_consensus.cc:348] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.535359 5974 raft_consensus.cc:374] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:40.535305 5977 log.cc:824] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:40.535437 5974 raft_consensus.cc:729] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b713ec6ca3443a48454881fbff6feff, State: Initialized, Role: FOLLOWER
I20260620 12:16:40.536147 5973 consensus_queue.cc:260] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.537649 5973 ts_tablet_manager.cc:1428] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Time spent starting tablet: real 0.018s user 0.010s sys 0.000s
I20260620 12:16:40.538344 5974 consensus_queue.cc:260] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.538933 5977 tablet_bootstrap.cc:492] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c: No bootstrap required, opened a new log
I20260620 12:16:40.539271 5977 ts_tablet_manager.cc:1397] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260620 12:16:40.539842 5974 ts_tablet_manager.cc:1428] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260620 12:16:40.540387 5940 heartbeater.cc:503] Master 127.24.141.62:32901 was elected leader, sending a full tablet report...
I20260620 12:16:40.545301 5806 heartbeater.cc:503] Master 127.24.141.62:32901 was elected leader, sending a full tablet report...
I20260620 12:16:40.545607 5977 raft_consensus.cc:348] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.546162 5977 raft_consensus.cc:374] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:40.546252 5977 raft_consensus.cc:729] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825f7a9c6c844b2696c499e85a81707c, State: Initialized, Role: FOLLOWER
I20260620 12:16:40.547184 5977 consensus_queue.cc:260] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.550506 5977 ts_tablet_manager.cc:1428] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 12:16:40.551041 5672 heartbeater.cc:503] Master 127.24.141.62:32901 was elected leader, sending a full tablet report...
I20260620 12:16:40.554235 5981 raft_consensus.cc:482] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:16:40.554615 5981 raft_consensus.cc:504] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.557778 5981 leader_election.cc:288] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b713ec6ca3443a48454881fbff6feff (127.24.141.2:35087), 28f05c39597e4144b2965a5f8fa8406e (127.24.141.3:43101)
I20260620 12:16:40.564009 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.557570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:42821 (local address 127.24.141.3:43101)
0620 12:16:40.557733 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:40.557738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:40.558294 (+ 556us) server_negotiation.cc:407] Connection header received
0620 12:16:40.558544 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:40.558548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:40.558596 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:40.558638 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:40.559111 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.559902 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.562221 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.562965 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.563105 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.563196 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:40.563736 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 12:16:40.563846 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:16:40.564078 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.557616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:46385 (local address 127.24.141.2:35087)
0620 12:16:40.557771 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:40.557777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:40.558162 (+ 385us) server_negotiation.cc:407] Connection header received
0620 12:16:40.558396 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:40.558400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:40.558455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:40.558504 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:40.559022 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.559886 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.562343 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.563025 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.563159 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.563229 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:40.563740 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 12:16:40.563882 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:16:40.564805 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.556980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:43101 (local address 127.24.141.1:42821)
0620 12:16:40.558173 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:40.558189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:40.558411 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:40.558784 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:40.558787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:40.558807 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:40.558978 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.558986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.560095 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.560099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.562019 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.562039 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.563341 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.563345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.563543 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.563589 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 12:16:40.564433 (+ 844us) client_negotiation.cc:240] Negotiation successful
0620 12:16:40.564554 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":250,"thread_start_us":243,"threads_started":1}
I20260620 12:16:40.565382 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.556957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:35087 (local address 127.24.141.1:46385)
0620 12:16:40.558047 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:40.558066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:40.558263 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:40.558665 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:40.558668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:40.558687 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:40.558866 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.558874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.560060 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.560063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.562122 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:40.562140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.563353 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.563357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:40.563527 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.563573 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 12:16:40.564824 (+ 1251us) client_negotiation.cc:240] Negotiation successful
0620 12:16:40.565138 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":108,"thread_start_us":283,"threads_started":1}
I20260620 12:16:40.565976 5894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4446f913c42e08348c410a231bddc" candidate_uuid: "825f7a9c6c844b2696c499e85a81707c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f05c39597e4144b2965a5f8fa8406e" is_pre_election: true
I20260620 12:16:40.566457 5894 raft_consensus.cc:2454] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825f7a9c6c844b2696c499e85a81707c in term 0.
I20260620 12:16:40.566615 5760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4446f913c42e08348c410a231bddc" candidate_uuid: "825f7a9c6c844b2696c499e85a81707c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b713ec6ca3443a48454881fbff6feff" is_pre_election: true
I20260620 12:16:40.567046 5760 raft_consensus.cc:2454] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825f7a9c6c844b2696c499e85a81707c in term 0.
I20260620 12:16:40.567265 5559 leader_election.cc:302] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f05c39597e4144b2965a5f8fa8406e, 825f7a9c6c844b2696c499e85a81707c; no voters:
I20260620 12:16:40.568109 5981 raft_consensus.cc:2781] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:16:40.568334 5981 raft_consensus.cc:482] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:16:40.568397 5981 raft_consensus.cc:3037] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:40.570020 5981 raft_consensus.cc:504] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.570693 5981 leader_election.cc:288] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [CANDIDATE]: Term 1 election: Requested vote from peers 3b713ec6ca3443a48454881fbff6feff (127.24.141.2:35087), 28f05c39597e4144b2965a5f8fa8406e (127.24.141.3:43101)
I20260620 12:16:40.571269 5760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4446f913c42e08348c410a231bddc" candidate_uuid: "825f7a9c6c844b2696c499e85a81707c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b713ec6ca3443a48454881fbff6feff"
I20260620 12:16:40.571437 5760 raft_consensus.cc:3037] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:40.571672 5894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4446f913c42e08348c410a231bddc" candidate_uuid: "825f7a9c6c844b2696c499e85a81707c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f05c39597e4144b2965a5f8fa8406e"
I20260620 12:16:40.571942 5894 raft_consensus.cc:3037] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:40.573006 5760 raft_consensus.cc:2454] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825f7a9c6c844b2696c499e85a81707c in term 1.
I20260620 12:16:40.573686 5561 leader_election.cc:302] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b713ec6ca3443a48454881fbff6feff, 825f7a9c6c844b2696c499e85a81707c; no voters:
I20260620 12:16:40.573938 5981 raft_consensus.cc:2781] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:16:40.574196 5981 raft_consensus.cc:686] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [term 1 LEADER]: Becoming Leader. State: Replica: 825f7a9c6c844b2696c499e85a81707c, State: Running, Role: LEADER
I20260620 12:16:40.574769 5894 raft_consensus.cc:2454] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825f7a9c6c844b2696c499e85a81707c in term 1.
I20260620 12:16:40.574791 5981 consensus_queue.cc:237] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } }
I20260620 12:16:40.581422 5474 catalog_manager.cc:5495] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c reported cstate change: term changed from 0 to 1, leader changed from <none> to 825f7a9c6c844b2696c499e85a81707c (127.24.141.1). New cstate: current_term: 1 leader_uuid: "825f7a9c6c844b2696c499e85a81707c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825f7a9c6c844b2696c499e85a81707c" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 44737 } health_report { overall_health: HEALTHY } } }
W20260620 12:16:40.583930 5941 tablet.cc:2369] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:16:40.585173 5673 tablet.cc:2369] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:16:40.626012 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.619399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.24.141.1:44737)
0620 12:16:40.619608 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:40.619615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:40.619796 (+ 181us) server_negotiation.cc:407] Connection header received
0620 12:16:40.620050 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:40.620054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:40.620151 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:40.620307 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:16:40.620660 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.621647 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.623277 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:40.623651 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:40.623788 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:40.624428 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:16:40.625376 (+ 948us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:16:40.625536 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:40.625675 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 12:16:40.625786 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:16:40.637259 5894 raft_consensus.cc:1261] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 1 FOLLOWER]: Refusing update from remote peer 825f7a9c6c844b2696c499e85a81707c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:40.637348 5760 raft_consensus.cc:1261] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 1 FOLLOWER]: Refusing update from remote peer 825f7a9c6c844b2696c499e85a81707c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:40.638616 5981 consensus_queue.cc:1035] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f05c39597e4144b2965a5f8fa8406e" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:16:40.639078 5984 consensus_queue.cc:1035] T 81b4446f913c42e08348c410a231bddc P 825f7a9c6c844b2696c499e85a81707c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b713ec6ca3443a48454881fbff6feff" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 35087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:16:40.660893 5989 mvcc.cc:205] Tried to move back new op lower bound from 7298899151397638144 to 7298899151162191872. Current Snapshot: MvccSnapshot[applied={T|T < 7298899151397638144}]
I20260620 12:16:40.660997 5988 mvcc.cc:205] Tried to move back new op lower bound from 7298899151397638144 to 7298899151162191872. Current Snapshot: MvccSnapshot[applied={T|T < 7298899151397638144}]
I20260620 12:16:40.674942 5471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:16:40.675257 5471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:16:40.679677 5471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df46d2f62f274b03a823e359711e94f5: Sending DeleteTablet for 3 replicas of tablet 81b4446f913c42e08348c410a231bddc
I20260620 12:16:40.681309 5874 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b4446f913c42e08348c410a231bddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:40 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:49312
I20260620 12:16:40.682364 5738 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b4446f913c42e08348c410a231bddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:40 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:35086
I20260620 12:16:40.684298 5605 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b4446f913c42e08348c410a231bddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:40 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:49240
I20260620 12:16:40.685431 5997 tablet_replica.cc:330] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: stopping tablet replica
I20260620 12:16:40.686308 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 5512
I20260620 12:16:40.687403 5997 raft_consensus.cc:2229] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:16:40.687897 5997 raft_consensus.cc:2258] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:40.688422 5998 tablet_replica.cc:330] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: stopping tablet replica
I20260620 12:16:40.688781 5998 raft_consensus.cc:2229] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:16:40.689204 5998 raft_consensus.cc:2258] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:40.691232 5997 ts_tablet_manager.cc:1905] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:16:40.695546 5997 ts_tablet_manager.cc:1918] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:16:40.695698 5997 log.cc:1196] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/81b4446f913c42e08348c410a231bddc
I20260620 12:16:40.696066 5997 ts_tablet_manager.cc:1939] T 81b4446f913c42e08348c410a231bddc P 28f05c39597e4144b2965a5f8fa8406e: Deleting consensus metadata
I20260620 12:16:40.697996 5458 catalog_manager.cc:4841] TS 28f05c39597e4144b2965a5f8fa8406e (127.24.141.3:43101): tablet 81b4446f913c42e08348c410a231bddc (table test-table [id=89c75921b0274577b27fff8c2975fd1d]) successfully deleted
I20260620 12:16:40.698988 5998 ts_tablet_manager.cc:1905] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:16:40.702519 5998 ts_tablet_manager.cc:1918] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:16:40.702803 5998 log.cc:1196] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/81b4446f913c42e08348c410a231bddc
I20260620 12:16:40.703234 5998 ts_tablet_manager.cc:1939] T 81b4446f913c42e08348c410a231bddc P 3b713ec6ca3443a48454881fbff6feff: Deleting consensus metadata
I20260620 12:16:40.705260 5460 catalog_manager.cc:4841] TS 3b713ec6ca3443a48454881fbff6feff (127.24.141.2:35087): tablet 81b4446f913c42e08348c410a231bddc (table test-table [id=89c75921b0274577b27fff8c2975fd1d]) successfully deleted
W20260620 12:16:40.706564 5459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:44737 (error 108)
I20260620 12:16:40.707147 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 5676
I20260620 12:16:40.715000 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:40.714448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:44737 (local address 127.0.0.1:49260)
0620 12:16:40.714717 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:40.714859 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:44737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 12:16:40.715062 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 5809
W20260620 12:16:40.715420 5459 catalog_manager.cc:4584] TS 825f7a9c6c844b2696c499e85a81707c (127.24.141.1:44737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81b4446f913c42e08348c410a231bddc: Network error: Client connection negotiation failed: client connection to 127.24.141.1:44737: connect: Connection refused (error 111)
I20260620 12:16:40.722867 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 5441
I20260620 12:16:40.730883 25140 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 12:16:41.077513 25140 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 12:16:41.077692 25140 mini_ranger.cc:66] Stopping Ranger...
I20260620 12:16:41.425516 25140 mini_ranger.cc:68] Stopped Ranger
2026-06-20 12:16:41.425 UTC [4507] LOG: received smart shutdown request
2026-06-20 12:16:41.430 UTC [4507] LOG: background worker "logical replication launcher" (PID 4516) exited with exit code 1
2026-06-20 12:16:41.430 UTC [4511] LOG: shutting down
2026-06-20 12:16:41.662 UTC [4507] LOG: database system is shut down
2026-06-20T12:16:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59021 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:16:41 dist-test-slave-rh5v krb5kdc[6009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:16:41 dist-test-slave-rh5v krb5kdc[6009](info): set up 2 sockets
Jun 20 12:16:41 dist-test-slave-rh5v krb5kdc[6009](info): commencing operation
krb5kdc: starting...
W20260620 12:16:42.773939 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:16:42 dist-test-slave-rh5v krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:16:42.924571 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:46059
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:46059,127.24.141.61:45693,127.24.141.60:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:43.067102 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:43.067819 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:43.067942 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:43.073235 6022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:43.073348 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:43.073395 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:43.073421 6022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:43.073455 6022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:43.080907 6022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:46059,127.24.141.61:45693,127.24.141.60:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:43.082674 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:43.084146 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:43.095417 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:43.095456 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:43.095597 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:43.096089 6022 server_base.cc:1034] running on GCE node
I20260620 12:16:43.096935 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:43.098623 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:43.099833 6022 hybrid_clock.cc:648] HybridClock initialized: now 1781957803099814 us; error 48 us; skew 500 ppm
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:43.103528 6022 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:16:43.104835 6022 webserver.cc:460] Webserver started at http://127.24.141.62:46697/ using document root <none> and password file <none>
I20260620 12:16:43.105634 6022 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:43.105717 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:43.105952 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:43.108651 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "a80055a5fcbc46fb95b29339c5e38d23"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "969e58fdedeb3b07fe136c38a587ea49"
server_key_iv: "b41cd50dbca01c524b2bb054871e0f3c"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.109416 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "a80055a5fcbc46fb95b29339c5e38d23"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "969e58fdedeb3b07fe136c38a587ea49"
server_key_iv: "b41cd50dbca01c524b2bb054871e0f3c"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.114847 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:16:43.118935 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:43.121176 6022 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:16:43.121387 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "a80055a5fcbc46fb95b29339c5e38d23"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "969e58fdedeb3b07fe136c38a587ea49"
server_key_iv: "b41cd50dbca01c524b2bb054871e0f3c"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.121590 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:43.137660 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:43.141945 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:43.142205 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:43.151808 6022 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:46059
I20260620 12:16:43.151842 6089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:46059 every 8 connection(s)
I20260620 12:16:43.153230 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:16:43.154372 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6022
I20260620 12:16:43.154564 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:16:43.154940 25140 external_mini_cluster.cc:1428] Setting key bcb472d7c7c1112dd43946128fadc063
I20260620 12:16:43.157841 6090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:43.163169 6090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.141.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.61@KRBTEST.COM" created.
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab.
Entry for principal kudu/127.24.141.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab.
I20260620 12:16:43.187996 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:45693
--webserver_interface=127.24.141.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab
--principal=kudu/127.24.141.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:46059,127.24.141.61:45693,127.24.141.60:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:16:43.188367 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.167984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.24.141.62:46059)
0620 12:16:43.168333 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.168345 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.168399 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:16:43.169764 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.169773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.170163 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.170495 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.171963 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.173149 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.174610 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.175237 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.175614 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.181484 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.181492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.181507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.181539 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.184236 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.184878 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.184884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.184885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.185013 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.185605 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.185609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.185611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.186196 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.186395 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.186842 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.187133 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":223,"spinlock_wait_cycles":8576,"thread_start_us":105,"threads_started":1}
I20260620 12:16:43.188369 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.166560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:46059 (local address 127.0.0.1:42952)
0620 12:16:43.167552 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.167681 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.169193 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.170584 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.170591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.171091 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.171642 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.171697 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.173303 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.173307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.174313 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.174328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.175573 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.175578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.175753 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.176703 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.176727 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.181224 (+ 4497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.184408 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.184417 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.184433 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.184764 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.185333 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.185337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.185339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.185499 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.186373 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.186381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.186546 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.186758 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.187215 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":132,"thread_start_us":213,"threads_started":1}
I20260620 12:16:43.192682 6090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:16:43.194018 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.193465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:47990)
0620 12:16:43.193797 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.193913 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
W20260620 12:16:43.194324 6038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
I20260620 12:16:43.195622 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.195352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48006)
0620 12:16:43.195490 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.195542 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 12:16:43.195771 6090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:45693: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
I20260620 12:16:43.245085 6090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } attempt: 1
I20260620 12:16:43.246344 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.245886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48020)
0620 12:16:43.246140 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.246231 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 12:16:43.247256 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.247015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48028)
0620 12:16:43.247130 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.247183 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 12:16:43.247390 6090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:45693: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
I20260620 12:16:43.315618 6090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } attempt: 2
I20260620 12:16:43.316799 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.316268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48042)
0620 12:16:43.316532 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.316653 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 12:16:43.317943 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.317641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48048)
0620 12:16:43.317783 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.317854 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 12:16:43.318152 6090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:45693: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
W20260620 12:16:43.335197 6096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:43.335662 6096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:43.335714 6096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:43.340919 6096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:43.341235 6096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:43.341285 6096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:43.341323 6096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:43.341358 6096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:43.348495 6096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:46059,127.24.141.61:45693,127.24.141.60:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab
--principal=kudu/127.24.141.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.141.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:43.350216 6096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:43.351711 6096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:43.364094 6102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:43.364075 6103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:43.364167 6096 server_base.cc:1034] running on GCE node
W20260620 12:16:43.364065 6105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:43.365221 6096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:43.366771 6096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:43.368046 6096 hybrid_clock.cc:648] HybridClock initialized: now 1781957803368023 us; error 51 us; skew 500 ppm
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:43.372288 6096 init.cc:377] Logged in from keytab as kudu/127.24.141.61@KRBTEST.COM (short username kudu)
I20260620 12:16:43.373719 6096 webserver.cc:460] Webserver started at http://127.24.141.61:39677/ using document root <none> and password file <none>
I20260620 12:16:43.374496 6096 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:43.374585 6096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:43.374832 6096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:43.377816 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data/instance:
uuid: "f3cd0b9264324d18ae538e198419936a"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "61b2801620efa022519f36ea73cbc195"
server_key_iv: "681255ca27514a5d9328b7f1fc58c78a"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.378544 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/wal/instance:
uuid: "f3cd0b9264324d18ae538e198419936a"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "61b2801620efa022519f36ea73cbc195"
server_key_iv: "681255ca27514a5d9328b7f1fc58c78a"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.384559 6096 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 12:16:43.389219 6112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:43.391276 6096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:16:43.391510 6096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/wal
uuid: "f3cd0b9264324d18ae538e198419936a"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "61b2801620efa022519f36ea73cbc195"
server_key_iv: "681255ca27514a5d9328b7f1fc58c78a"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.391655 6096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:43.409379 6090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } attempt: 3
I20260620 12:16:43.410634 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.410205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48056)
0620 12:16:43.410392 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.410514 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 12:16:43.411751 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.411439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48060)
0620 12:16:43.411604 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.411671 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 12:16:43.411944 6090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:45693: Network error: Client connection negotiation failed: client connection to 127.24.141.61:45693: connect: Connection refused (error 111)
I20260620 12:16:43.419777 6096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:43.424515 6096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:43.424764 6096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:43.433954 6096 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.61:45693
I20260620 12:16:43.433964 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.61:45693 every 8 connection(s)
I20260620 12:16:43.435230 6096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
I20260620 12:16:43.439036 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6096
I20260620 12:16:43.439298 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-1/wal/instance
I20260620 12:16:43.439627 25140 external_mini_cluster.cc:1428] Setting key 4b98aa3c0ac58a087bb51cc059e1ebbf
I20260620 12:16:43.439617 6165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:43.445865 6165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.141.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.60@KRBTEST.COM" created.
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:43.467933 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.448788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.24.141.62:46059)
0620 12:16:43.449032 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.449038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.450269 (+ 1231us) server_negotiation.cc:407] Connection header received
0620 12:16:43.451663 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.451668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.451767 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.451902 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.453747 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.454524 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.456942 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.457314 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.457436 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.461384 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.461391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.461394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.461457 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.463436 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.464691 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.464697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.464723 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.464908 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.465635 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.465639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.465641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.465999 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.466131 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.467288 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.467635 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 12:16:43.469486 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.449294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:46059 (local address 127.0.0.1:42966)
0620 12:16:43.449988 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.450072 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.451291 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.452175 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.452186 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.452687 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.453472 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.453497 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.454741 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.454745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.456676 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.456702 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.457541 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.457546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.457808 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.458612 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.458642 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.460999 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.463772 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.463785 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.463815 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.464447 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.465127 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.465132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.465134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.465452 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.466179 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.466188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.466754 (+ 566us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.467094 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.468175 (+ 1081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":187,"threads_started":1}
I20260620 12:16:43.472193 6165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab.
Entry for principal kudu/127.24.141.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab.
I20260620 12:16:43.477558 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:39823
--webserver_interface=127.24.141.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab
--principal=kudu/127.24.141.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:46059,127.24.141.61:45693,127.24.141.60:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:16:43.484979 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.473038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48070)
0620 12:16:43.473283 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.473301 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.473448 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.474550 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.474554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.474787 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.474950 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.474959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.476302 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.476308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.477031 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.477049 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.478320 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.478324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.478453 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.479141 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.479156 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.480759 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.483193 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.483197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.483200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.483419 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.483709 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.483712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.483714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.483792 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.484341 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.484344 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.484437 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.484543 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.484727 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 12:16:43.484999 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.473399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.24.141.61:45693)
0620 12:16:43.473910 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.473922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.473953 (+ 31us) server_negotiation.cc:407] Connection header received
0620 12:16:43.474022 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.474031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.474321 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.474503 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.475127 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.476065 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.477234 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.478087 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.478256 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.480966 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.480988 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.481012 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.481041 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.483036 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.483532 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.483539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.483541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.483622 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.483871 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.483874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.483876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.484231 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.484382 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.484655 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.484809 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":217,"threads_started":1}
I20260620 12:16:43.488642 6165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:16:43.489959 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.489390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38360)
0620 12:16:43.489635 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.489801 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 12:16:43.490439 6114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
I20260620 12:16:43.491906 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.491523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38372)
0620 12:16:43.491695 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.491792 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 12:16:43.492172 6165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:39823: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
I20260620 12:16:43.541564 6165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } attempt: 1
I20260620 12:16:43.542858 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.542391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38376)
0620 12:16:43.542642 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.542749 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 12:16:43.543853 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.543558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38392)
0620 12:16:43.543698 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.543773 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 12:16:43.544046 6165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:39823: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
I20260620 12:16:43.555217 6090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } attempt: 4
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:16:43.569510 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.556408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.24.141.61:45693)
0620 12:16:43.556738 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.556743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.556774 (+ 31us) server_negotiation.cc:407] Connection header received
0620 12:16:43.556842 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.556846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.556913 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.557035 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.557799 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.558671 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.560435 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.560958 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.561176 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.564929 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.564934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.564936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.564985 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.566669 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.567462 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.567467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.567468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.567583 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.568163 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.568167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.568169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.568454 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.568561 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.569170 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.569338 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260620 12:16:43.569519 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.556209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:48076)
0620 12:16:43.556525 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.556550 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.556718 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.557192 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.557195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.557444 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.557622 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.557630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.558889 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.558895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.560202 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.560225 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.561256 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.561260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.561406 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.562074 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.562089 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.564432 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.566901 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.566907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.566909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.567265 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.567689 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.567692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.567694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.567810 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.568610 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.568615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.568842 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.569008 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.569286 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260620 12:16:43.570721 6090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:16:43.571847 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.571438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38396)
0620 12:16:43.571654 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.571739 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 12:16:43.572718 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.572469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38400)
0620 12:16:43.572588 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.572644 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 12:16:43.572862 6090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:39823: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
I20260620 12:16:43.612293 6165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } attempt: 2
I20260620 12:16:43.613548 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.612936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38414)
0620 12:16:43.613247 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.613356 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 12:16:43.614733 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.614392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38426)
0620 12:16:43.614568 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.614643 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 12:16:43.614920 6165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:39823: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
W20260620 12:16:43.629930 6171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:43.630234 6171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:43.630277 6171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 12:16:43.632093 6090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } attempt: 1
I20260620 12:16:43.633301 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.632858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38436)
0620 12:16:43.633096 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.633177 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 12:16:43.634545 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.634229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38442)
0620 12:16:43.634356 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.634424 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 12:16:43.634720 6090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:39823: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
W20260620 12:16:43.635588 6171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:43.635685 6171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:43.635705 6171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:43.635722 6171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:43.635741 6171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:43.641894 6171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:46059,127.24.141.61:45693,127.24.141.60:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab
--principal=kudu/127.24.141.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:39823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.141.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:43.643085 6171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:43.644224 6171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:43.654577 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:43.654587 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:43.654659 6171 server_base.cc:1034] running on GCE node
W20260620 12:16:43.654587 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:43.655843 6171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:43.657500 6171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:43.658823 6171 hybrid_clock.cc:648] HybridClock initialized: now 1781957803658778 us; error 91 us; skew 500 ppm
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:43.662701 6171 init.cc:377] Logged in from keytab as kudu/127.24.141.60@KRBTEST.COM (short username kudu)
I20260620 12:16:43.664079 6171 webserver.cc:460] Webserver started at http://127.24.141.60:42539/ using document root <none> and password file <none>
I20260620 12:16:43.664844 6171 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:43.664922 6171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:43.665236 6171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:43.668102 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data/instance:
uuid: "91c0f53ee8cd4fff889dd6c8f840da0d"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "0390dff86d96b0c4e02d06f3910ac9f7"
server_key_iv: "6f180c47db31dd76fb505242651e70f1"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.668871 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/wal/instance:
uuid: "91c0f53ee8cd4fff889dd6c8f840da0d"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "0390dff86d96b0c4e02d06f3910ac9f7"
server_key_iv: "6f180c47db31dd76fb505242651e70f1"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.674412 6171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:16:43.677985 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:43.679869 6171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:16:43.680094 6171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/wal
uuid: "91c0f53ee8cd4fff889dd6c8f840da0d"
format_stamp: "Formatted at 2026-06-20 12:16:43 on dist-test-slave-rh5v"
server_key: "0390dff86d96b0c4e02d06f3910ac9f7"
server_key_iv: "6f180c47db31dd76fb505242651e70f1"
server_key_version: "encryptionkey@0"
I20260620 12:16:43.680263 6171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:43.701926 6090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } attempt: 2
I20260620 12:16:43.703154 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.702699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38446)
0620 12:16:43.702951 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.703043 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 12:16:43.704329 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.703912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38454)
0620 12:16:43.704156 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.704237 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 12:16:43.704504 6090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:39823: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
I20260620 12:16:43.706193 6165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } attempt: 3
I20260620 12:16:43.707446 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.706869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38468)
0620 12:16:43.707108 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.707237 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 12:16:43.708644 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.708240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:38480)
0620 12:16:43.708472 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.708537 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260620 12:16:43.708801 6165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:39823: Network error: Client connection negotiation failed: client connection to 127.24.141.60:39823: connect: Connection refused (error 111)
I20260620 12:16:43.722193 6171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:43.726274 6171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:43.726522 6171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:43.735697 6171 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.60:39823
I20260620 12:16:43.735791 6239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.60:39823 every 8 connection(s)
I20260620 12:16:43.737006 6171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
I20260620 12:16:43.738099 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6171
I20260620 12:16:43.738243 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957618728254-25140-0/minicluster-data/master-2/wal/instance
I20260620 12:16:43.738583 25140 external_mini_cluster.cc:1428] Setting key 29baf5d247bc9aeeca072cd9bb20e3dd
I20260620 12:16:43.741133 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:43.746307 6240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:43.757938 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.740352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.24.141.60:39823)
0620 12:16:43.740882 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.740901 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.740950 (+ 49us) server_negotiation.cc:407] Connection header received
0620 12:16:43.741825 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.741849 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.742305 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.742703 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.743786 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.745148 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.746557 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.747324 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.747632 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.750684 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.750693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.750708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.750753 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.753549 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.754331 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.754339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.754353 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.754506 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.754813 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.754816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.754818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.755657 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.756023 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.756418 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.756815 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":155,"threads_started":1}
I20260620 12:16:43.762408 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.748781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:46059 (local address 127.0.0.1:42980)
0620 12:16:43.749570 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.749644 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.749836 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.750238 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.750244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.750816 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.751042 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.751052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.752301 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.752307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.753229 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.753257 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.753918 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.753922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.754066 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.754659 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.754672 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.756539 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.758911 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.758923 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.758929 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.759281 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.759732 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.759735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.759737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.759851 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.760805 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.760814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.761099 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.761883 (+ 784us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.762170 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":328,"threads_started":1}
I20260620 12:16:43.762665 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.749020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.24.141.62:46059)
0620 12:16:43.749307 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.749314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.749775 (+ 461us) server_negotiation.cc:407] Connection header received
0620 12:16:43.749954 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.749959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.750035 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.750151 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.751253 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.752026 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.753434 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.753788 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.753898 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.756802 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.756806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.756809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.756848 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.758710 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.759409 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.759413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.759415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.759541 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.760023 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.760028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.760029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.760518 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.760800 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.762082 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.762317 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:16:43.764232 6240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:16:43.773895 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.762815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.24.141.62:46059)
0620 12:16:43.762927 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.762931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.762961 (+ 30us) server_negotiation.cc:407] Connection header received
0620 12:16:43.763010 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.763013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.763067 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.763163 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.764242 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.764853 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.766076 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.766479 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.766617 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.769739 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.769743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.769745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.769789 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.771692 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.772322 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.772325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.772327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.772432 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.772714 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.772716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.772718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.772984 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.773328 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.773510 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.773674 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 12:16:43.773999 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.765054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.24.141.61:45693)
0620 12:16:43.765223 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.765227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.765428 (+ 201us) server_negotiation.cc:407] Connection header received
0620 12:16:43.765499 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.765501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.765556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.765643 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.766213 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.766922 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.768019 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.768257 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.768355 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.770592 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.770597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.770601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.770638 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.772006 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.772501 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.772505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.772507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.772589 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.772891 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.772894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.772896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.773200 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.773292 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.773717 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.773841 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:16:43.774549 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.764933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:45693 (local address 127.0.0.1:33798)
0620 12:16:43.765269 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.765287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.765410 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.765686 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.765689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.765925 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.766076 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.766084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.767067 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.767071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.767849 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.767864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.768395 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.768398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.768499 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.769001 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.769018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.770397 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.772152 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.772157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.772158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.772403 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.772698 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.772702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.772704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.772797 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.773365 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.773381 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.773517 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.774236 (+ 719us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.774408 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260620 12:16:43.775637 6240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:16:43.785627 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.776326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:52402)
0620 12:16:43.776665 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.776680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.776792 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.777128 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.777132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.777292 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.777405 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.777411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.778263 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.778266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.778954 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.778963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.779439 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.779443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.779522 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.779944 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.779955 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.781487 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.783599 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.783601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.783603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.783819 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.784139 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.784144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.784146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.784248 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.784759 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.784764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.784953 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.785143 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.785370 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260620 12:16:43.785669 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.776352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.24.141.60:39823)
0620 12:16:43.776585 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.776590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.776748 (+ 158us) server_negotiation.cc:407] Connection header received
0620 12:16:43.776890 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.776892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.776962 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.777046 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.777518 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.778132 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.779072 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.779322 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.779414 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.781632 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.781636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.781639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.781664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.783424 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.783918 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.783922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.783925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.784019 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.784338 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.784342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.784344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.784641 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.784749 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.785211 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.785505 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":139}
I20260620 12:16:43.786562 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.775629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.24.141.61:45693)
0620 12:16:43.775760 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.775765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.775779 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:16:43.776070 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.776074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.776129 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.776201 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.777037 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.777729 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.778709 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.778943 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.779027 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.782209 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.782214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.782217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.782260 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.784057 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.784693 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.784701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.784703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.784892 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.785411 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.785415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.785416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.785744 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.786060 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.786233 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.786392 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:16:43.791247 6240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d: Bootstrap starting.
I20260620 12:16:43.792882 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:16:43.794900 6240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:43.796281 6240 log.cc:824] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:43.799532 6240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d: No bootstrap required, opened a new log
I20260620 12:16:43.804693 6090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } attempt: 3
I20260620 12:16:43.804786 6240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.805253 6240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:43.805323 6240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c0f53ee8cd4fff889dd6c8f840da0d, State: Initialized, Role: FOLLOWER
I20260620 12:16:43.806089 6240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.807633 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.796198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.24.141.62:46059)
0620 12:16:43.796362 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.796367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.797358 (+ 991us) server_negotiation.cc:407] Connection header received
0620 12:16:43.797445 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.797449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.797527 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.797658 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.798496 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.799030 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.800155 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.800392 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.800486 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.802262 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.802266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.802269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.802300 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.804654 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.805631 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.805636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.805638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.805777 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.806241 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.806245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.806246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.806471 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.806587 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.807287 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.807438 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:16:43.807657 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.795633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.24.141.61:45693)
0620 12:16:43.795819 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.795825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.795901 (+ 76us) server_negotiation.cc:407] Connection header received
0620 12:16:43.796133 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.796137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.796205 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.796292 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.797422 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.798186 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.799491 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.799863 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.799991 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.802471 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.802475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.802478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.802510 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.804654 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.805632 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.805636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.805638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.805777 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.806103 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.806107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.806109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.806414 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.806586 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.807356 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.807512 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 12:16:43.808180 6240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
W20260620 12:16:43.815901 6265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:16:43.816030 6265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:16:43.813470 6171 master_runner.cc:186] Error getting master registration for 127.24.141.62:46059: OK, instance_id { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" instance_seqno: 1781957803148860 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:16:43.817385 6171 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:16:43.811401 6254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.817612 6254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:16:43.820282 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.796300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.24.141.60:39823)
0620 12:16:43.796427 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.796432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.805563 (+ 9131us) server_negotiation.cc:407] Connection header received
0620 12:16:43.806739 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.806743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.806819 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.806893 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.810887 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.811721 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.812992 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.813419 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.813513 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.815871 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.815893 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.815899 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.815956 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.818136 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.818727 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.818731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.818733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.818863 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.819277 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.819283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.819285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.819541 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.819672 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.819966 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.820109 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:16:43.820761 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.805402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:52426)
0620 12:16:43.805830 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.805845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.805932 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.807114 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.807118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.807330 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.807456 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.807464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.809289 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.809294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.810173 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.810189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.810910 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.810914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.811016 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.812048 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.812069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.813855 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.818656 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.818660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.818663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.819082 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.819454 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.819458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.819471 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.819603 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.820122 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.820126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.820255 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.820342 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.820569 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337}
I20260620 12:16:43.821497 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.806114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.24.141.60:39823)
0620 12:16:43.806849 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.806859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.806873 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:16:43.806908 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.806911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.806955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.807578 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.808154 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.809126 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.810360 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.810702 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.810811 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.814079 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.814084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.814087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.814116 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.818379 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.819205 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.819208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.819210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.819305 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.819733 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.819736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.819738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.819998 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.821025 (+ 1027us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.821153 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.821310 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":160,"threads_started":1}
I20260620 12:16:43.825112 6090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: Bootstrap starting.
I20260620 12:16:43.829365 6090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:43.830341 6090 log.cc:824] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:43.834787 6090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: No bootstrap required, opened a new log
I20260620 12:16:43.840282 6090 raft_consensus.cc:348] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.840683 6090 raft_consensus.cc:374] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:43.840739 6090 raft_consensus.cc:729] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80055a5fcbc46fb95b29339c5e38d23, State: Initialized, Role: FOLLOWER
I20260620 12:16:43.841440 6090 consensus_queue.cc:260] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.842293 6268 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.842540 6268 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:16:43.843541 6090 sys_catalog.cc:564] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:43.848081 6254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:16:43.848349 6254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.849841 6254 leader_election.cc:288] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80055a5fcbc46fb95b29339c5e38d23 (127.24.141.62:46059), f3cd0b9264324d18ae538e198419936a (127.24.141.61:45693)
I20260620 12:16:43.851881 6064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80055a5fcbc46fb95b29339c5e38d23" is_pre_election: true
I20260620 12:16:43.851958 6165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } attempt: 4
I20260620 12:16:43.852361 6064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c0f53ee8cd4fff889dd6c8f840da0d in term 0.
I20260620 12:16:43.852985 6191 leader_election.cc:302] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c0f53ee8cd4fff889dd6c8f840da0d, a80055a5fcbc46fb95b29339c5e38d23; no voters:
I20260620 12:16:43.853453 6254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:16:43.853550 6254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:16:43.853600 6254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:43.855015 6254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.855638 6254 leader_election.cc:288] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [CANDIDATE]: Term 1 election: Requested vote from peers a80055a5fcbc46fb95b29339c5e38d23 (127.24.141.62:46059), f3cd0b9264324d18ae538e198419936a (127.24.141.61:45693)
I20260620 12:16:43.852265 6022 master_runner.cc:186] Error getting master registration for 127.24.141.61:45693: OK, instance_id { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" instance_seqno: 1781957803431136 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:16:43.856773 6022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:16:43.857926 6064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80055a5fcbc46fb95b29339c5e38d23"
I20260620 12:16:43.858105 6064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [term 0 FOLLOWER]: Advancing to term 1
W20260620 12:16:43.858469 6279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:16:43.858592 6279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:16:43.859944 6064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c0f53ee8cd4fff889dd6c8f840da0d in term 1.
I20260620 12:16:43.860432 6191 leader_election.cc:302] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c0f53ee8cd4fff889dd6c8f840da0d, a80055a5fcbc46fb95b29339c5e38d23; no voters:
I20260620 12:16:43.860777 6254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:16:43.861148 6254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [term 1 LEADER]: Becoming Leader. State: Replica: 91c0f53ee8cd4fff889dd6c8f840da0d, State: Running, Role: LEADER
I20260620 12:16:43.861686 6254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.864609 6254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c0f53ee8cd4fff889dd6c8f840da0d. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
Jun 20 12:16:43 dist-test-slave-rh5v krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
I20260620 12:16:43.869549 6254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:43.870401 6283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:16:43.871865 6283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:16:43.882395 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.857263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:39823 (local address 127.0.0.1:52436)
0620 12:16:43.857958 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:43.857973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:43.858067 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:43.859145 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:43.859149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:43.859403 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:43.859525 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.859532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.860991 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.860996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.862092 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:43.862109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.863841 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.863846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:43.863955 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.865350 (+ 1395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:43.865371 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:43.868600 (+ 3229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:43.876311 (+ 7711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.876317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.876320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.876728 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.880903 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:43.880908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:43.880911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:43.881103 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:43.881673 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:43.881677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:43.881820 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 12:16:43.881911 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:16:43.882163 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600}
I20260620 12:16:43.883201 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:43.858615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.24.141.60:39823)
0620 12:16:43.858785 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:43.858789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:43.858804 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:16:43.858848 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:43.858851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:43.858905 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:43.858981 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:43.859712 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.860606 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.862677 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:43.863475 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:43.867694 (+ 4219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:43.869027 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:43.869034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:43.869038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:43.869109 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:43.876021 (+ 6912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.880549 (+ 4528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.880553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.880556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.880766 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:43.881236 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:43.881240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:43.881241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:43.881553 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:43.882673 (+ 1120us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:43.882789 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 12:16:43.882973 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:16:43.888780 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: Bootstrap starting.
I20260620 12:16:43.889484 6064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [term 1 FOLLOWER]: Refusing update from remote peer 91c0f53ee8cd4fff889dd6c8f840da0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:43.890655 6281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:16:43.894832 6043 tablet.cc:2369] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:16:43.896354 6165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:43.897886 6165 log.cc:824] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:43.901197 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: No bootstrap required, opened a new log
I20260620 12:16:43.906597 6268 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c0f53ee8cd4fff889dd6c8f840da0d. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.906790 6268 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:16:43.909495 6254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.909713 6254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:43.910131 6284 mvcc.cc:205] Tried to move back new op lower bound from 7298899164709449728 to 7298899164622569472. Current Snapshot: MvccSnapshot[applied={T|T < 7298899164709449728}]
I20260620 12:16:43.910264 6268 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.910507 6268 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:16:43.911124 6165 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.911464 6165 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:43.911528 6165 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3cd0b9264324d18ae538e198419936a, State: Initialized, Role: FOLLOWER
I20260620 12:16:43.912339 6165 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } }
I20260620 12:16:43.914525 6165 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:43.914619 6138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3cd0b9264324d18ae538e198419936a"
I20260620 12:16:43.915078 6138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:43.916900 6138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c0f53ee8cd4fff889dd6c8f840da0d in term 1.
I20260620 12:16:43.917375 6139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3cd0b9264324d18ae538e198419936a" is_pre_election: true
I20260620 12:16:43.917549 6139 raft_consensus.cc:2362] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 91c0f53ee8cd4fff889dd6c8f840da0d in term 1. Re-sending same reply.
I20260620 12:16:43.918135 6137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [term 1 FOLLOWER]: Refusing update from remote peer 91c0f53ee8cd4fff889dd6c8f840da0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:43.918841 6254 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:16:43.919708 6285 mvcc.cc:205] Tried to move back new op lower bound from 7298899164709449728 to 7298899164622569472. Current Snapshot: MvccSnapshot[applied={T|T < 7298899164709449728}]
I20260620 12:16:43.923403 6283 catalog_manager.cc:1347] Generated new cluster ID: 6b337ceebe294ebd8534c7ee8b39a8e6
I20260620 12:16:43.923508 6283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:16:43.930367 6289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.930581 6289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:16:43.930696 6289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c0f53ee8cd4fff889dd6c8f840da0d. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.930773 6289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:16:43.930863 6289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.930936 6289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 12:16:43.927230 6118 tablet.cc:2369] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:16:43.935027 6301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:16:43.935176 6301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:16:43.959614 6283 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:16:43.961416 6283 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:16:43.973016 6281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80055a5fcbc46fb95b29339c5e38d23" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 46059 } } peers { permanent_uuid: "f3cd0b9264324d18ae538e198419936a" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 45693 } } peers { permanent_uuid: "91c0f53ee8cd4fff889dd6c8f840da0d" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 39823 } } }
I20260620 12:16:43.973268 6281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:43.974313 6303 mvcc.cc:205] Tried to move back new op lower bound from 7298899164709449728 to 7298899164622569472. Current Snapshot: MvccSnapshot[applied={T|T < 7298899164709449728}]
I20260620 12:16:44.015708 6283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91c0f53ee8cd4fff889dd6c8f840da0d: Generated new TSK 0
I20260620 12:16:44.017108 6283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:16:44.860154 6279 catalog_manager.cc:1259] Loaded cluster ID: 6b337ceebe294ebd8534c7ee8b39a8e6
I20260620 12:16:44.860265 6279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: loading cluster ID for follower catalog manager: success
I20260620 12:16:44.862255 6279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: acquiring CA information for follower catalog manager: success
I20260620 12:16:44.863798 6279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a80055a5fcbc46fb95b29339c5e38d23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:16:44.937162 6301 catalog_manager.cc:1259] Loaded cluster ID: 6b337ceebe294ebd8534c7ee8b39a8e6
I20260620 12:16:44.937292 6301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: loading cluster ID for follower catalog manager: success
I20260620 12:16:44.939386 6301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: acquiring CA information for follower catalog manager: success
I20260620 12:16:44.941107 6301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3cd0b9264324d18ae538e198419936a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:16:45.100616 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6022
I20260620 12:16:45.109344 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6096
I20260620 12:16:45.117017 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6171
2026-06-20T12:16:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3343 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:16:45 dist-test-slave-rh5v krb5kdc[6314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:16:45 dist-test-slave-rh5v krb5kdc[6314](info): set up 2 sockets
Jun 20 12:16:45 dist-test-slave-rh5v krb5kdc[6314](info): commencing operation
krb5kdc: starting...
W20260620 12:16:46.105860 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:16:46 dist-test-slave-rh5v krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:16:46.268379 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:42461
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:42461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:46.407565 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:46.408059 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:46.408150 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:46.412993 6327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:46.413148 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:46.413182 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:46.413203 6327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:46.413223 6327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:46.420686 6327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:42461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:46.422039 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:46.423240 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:46.434247 6332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:46.434191 6333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:46.434494 6327 server_base.cc:1034] running on GCE node
W20260620 12:16:46.434191 6335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:46.435525 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:46.437160 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:46.438444 6327 hybrid_clock.cc:648] HybridClock initialized: now 1781957806438421 us; error 52 us; skew 500 ppm
Jun 20 12:16:46 dist-test-slave-rh5v krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957806, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:46.442301 6327 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:16:46.444036 6327 webserver.cc:460] Webserver started at http://127.24.141.62:43329/ using document root <none> and password file <none>
I20260620 12:16:46.444808 6327 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:46.444900 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:46.445212 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:46.447757 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5"
format_stamp: "Formatted at 2026-06-20 12:16:46 on dist-test-slave-rh5v"
server_key: "7d3793370cde38ad44119a2630c6d477"
server_key_iv: "4401a6dc8912e4864e06b5771809dbd7"
server_key_version: "encryptionkey@0"
I20260620 12:16:46.448851 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5"
format_stamp: "Formatted at 2026-06-20 12:16:46 on dist-test-slave-rh5v"
server_key: "7d3793370cde38ad44119a2630c6d477"
server_key_iv: "4401a6dc8912e4864e06b5771809dbd7"
server_key_version: "encryptionkey@0"
I20260620 12:16:46.455154 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 12:16:46.459153 6342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:46.461370 6327 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:16:46.461580 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5"
format_stamp: "Formatted at 2026-06-20 12:16:46 on dist-test-slave-rh5v"
server_key: "7d3793370cde38ad44119a2630c6d477"
server_key_iv: "4401a6dc8912e4864e06b5771809dbd7"
server_key_version: "encryptionkey@0"
I20260620 12:16:46.461742 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:46.492800 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:46.497594 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:46.497844 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:46.506621 6327 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:42461
I20260620 12:16:46.506665 6394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:42461 every 8 connection(s)
I20260620 12:16:46.507793 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:16:46.509972 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6327
I20260620 12:16:46.510102 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:16:46.510403 25140 external_mini_cluster.cc:1428] Setting key 571db91d26f412876e3bb00c1aecfe5d
I20260620 12:16:46.512272 6395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:46.521716 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5: Bootstrap starting.
Jun 20 12:16:46 dist-test-slave-rh5v krb5kdc[6314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:46.525491 6395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:46.526516 6395 log.cc:824] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:46.529405 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5: No bootstrap required, opened a new log
I20260620 12:16:46.530494 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:46.512184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.24.141.62:42461)
0620 12:16:46.512946 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:46.512971 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:46.513028 (+ 57us) server_negotiation.cc:407] Connection header received
0620 12:16:46.514117 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:46.514154 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:46.514719 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:46.515199 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:46.516190 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:46.517895 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:46.519091 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:46.519926 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:46.520158 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:46.523933 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:46.523943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:46.523967 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:46.524022 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:46.526934 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:46.527545 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:46.527551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:46.527558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:46.527726 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:46.528017 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:46.528021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:46.528022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:46.528563 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:46.528766 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:46.529159 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 12:16:46.529558 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260620 12:16:46.534637 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:16:46.534408 6395 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42461 } }
I20260620 12:16:46.534791 6395 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:46.534871 6395 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2923aa1ac1a4efb92ddb264fa6ff6b5, State: Initialized, Role: FOLLOWER
I20260620 12:16:46.535440 6395 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42461 } }
I20260620 12:16:46.535616 6395 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:16:46.535689 6395 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:16:46.535817 6395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:46.537453 6395 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42461 } }
I20260620 12:16:46.537923 6395 leader_election.cc:302] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2923aa1ac1a4efb92ddb264fa6ff6b5; no voters:
I20260620 12:16:46.538341 6395 leader_election.cc:288] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:16:46.538717 6400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:16:46.539225 6400 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [term 1 LEADER]: Becoming Leader. State: Replica: a2923aa1ac1a4efb92ddb264fa6ff6b5, State: Running, Role: LEADER
I20260620 12:16:46.539678 6400 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42461 } }
I20260620 12:16:46.540007 6395 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:46.541872 6402 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2923aa1ac1a4efb92ddb264fa6ff6b5. Latest consensus state: current_term: 1 leader_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42461 } } }
I20260620 12:16:46.541864 6401 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2923aa1ac1a4efb92ddb264fa6ff6b5" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42461 } } }
I20260620 12:16:46.542112 6401 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:46.542112 6402 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5 [sys.catalog]: This master's current role is: LEADER
W20260620 12:16:46.546609 6415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:16:46.546783 6415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:16:46.546939 6414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:16:46.548117 6414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:16:46.555421 6414 catalog_manager.cc:1347] Generated new cluster ID: 34a79297424a4a2cbacb4a178d444f20
I20260620 12:16:46.555523 6414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:16:46.565267 6414 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:16:46.566928 6414 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:16:46.574970 6414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2923aa1ac1a4efb92ddb264fa6ff6b5: Generated new TSK 0
I20260620 12:16:46.575891 6414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:16:46.593367 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6327
2026-06-20T12:16:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1474 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:16:46 dist-test-slave-rh5v krb5kdc[6422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:16:46 dist-test-slave-rh5v krb5kdc[6422](info): set up 2 sockets
Jun 20 12:16:46 dist-test-slave-rh5v krb5kdc[6422](info): commencing operation
krb5kdc: starting...
W20260620 12:16:47.570211 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:16:47 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:16:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:16:47.717905 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45297
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:47.862424 6435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:47.862890 6435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:47.862957 6435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:47.867795 6435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:47.867916 6435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:47.867947 6435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:47.867969 6435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:47.867990 6435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:47.874848 6435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:47.876515 6435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:47.877987 6435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:47.889112 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:47.889118 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:47.889366 6435 server_base.cc:1034] running on GCE node
W20260620 12:16:47.889201 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:47.890419 6435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:47.892133 6435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:47.893359 6435 hybrid_clock.cc:648] HybridClock initialized: now 1781957807893323 us; error 66 us; skew 500 ppm
Jun 20 12:16:47 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957807, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:47.901471 6435 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:16:47.902850 6435 webserver.cc:460] Webserver started at http://127.24.141.62:42385/ using document root <none> and password file <none>
I20260620 12:16:47.903625 6435 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:47.903717 6435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:47.903959 6435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:47.908438 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "433a532ae71a48c8a2f56fa143752d2a"
format_stamp: "Formatted at 2026-06-20 12:16:47 on dist-test-slave-rh5v"
server_key: "b1ba99d1daa9441d94738e5f843e6c1e"
server_key_iv: "e2b89f221548afedb73f1e5040274618"
server_key_version: "encryptionkey@0"
I20260620 12:16:47.911450 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "433a532ae71a48c8a2f56fa143752d2a"
format_stamp: "Formatted at 2026-06-20 12:16:47 on dist-test-slave-rh5v"
server_key: "b1ba99d1daa9441d94738e5f843e6c1e"
server_key_iv: "e2b89f221548afedb73f1e5040274618"
server_key_version: "encryptionkey@0"
I20260620 12:16:47.921687 6435 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.004s
I20260620 12:16:47.927682 6450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:47.929991 6435 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 12:16:47.930235 6435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "433a532ae71a48c8a2f56fa143752d2a"
format_stamp: "Formatted at 2026-06-20 12:16:47 on dist-test-slave-rh5v"
server_key: "b1ba99d1daa9441d94738e5f843e6c1e"
server_key_iv: "e2b89f221548afedb73f1e5040274618"
server_key_version: "encryptionkey@0"
I20260620 12:16:47.930418 6435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:47.950842 6435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:47.955667 6435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:47.955894 6435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:47.966806 6435 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:45297
I20260620 12:16:47.967231 6502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:45297 every 8 connection(s)
I20260620 12:16:47.968282 6435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:16:47.968575 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6435
I20260620 12:16:47.968711 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:16:47.969089 25140 external_mini_cluster.cc:1428] Setting key 9b90b3fbf0836e37be59a475ae144634
I20260620 12:16:47.972540 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 12:16:47 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:47.980715 6503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Bootstrap starting.
I20260620 12:16:47.984015 6503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:47.985023 6503 log.cc:824] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:47.986359 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:47.970890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.24.141.62:45297)
0620 12:16:47.971678 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:47.971699 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:47.971757 (+ 58us) server_negotiation.cc:407] Connection header received
0620 12:16:47.972578 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:47.972598 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:47.973168 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:47.973611 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:47.974565 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:47.975725 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:47.976687 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:47.977336 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:47.977527 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:47.980280 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:47.980287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:47.980308 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:47.980355 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:47.982692 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:47.983278 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:47.983284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:47.983291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:47.983440 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:47.983758 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:47.983760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:47.983762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:47.984330 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:47.984534 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:47.984885 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 12:16:47.985316 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":215,"threads_started":1}
I20260620 12:16:47.988020 6503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: No bootstrap required, opened a new log
I20260620 12:16:47.992756 6503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:47.993206 6503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:47.993287 6503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a532ae71a48c8a2f56fa143752d2a, State: Initialized, Role: FOLLOWER
I20260620 12:16:47.993992 6503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:47.994184 6503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:16:47.994274 6503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:16:47.994402 6503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:47.995675 6503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:47.996133 6503 leader_election.cc:302] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 433a532ae71a48c8a2f56fa143752d2a; no voters:
I20260620 12:16:47.996584 6503 leader_election.cc:288] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:16:47.996716 6508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:16:47.996927 6508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 LEADER]: Becoming Leader. State: Replica: 433a532ae71a48c8a2f56fa143752d2a, State: Running, Role: LEADER
I20260620 12:16:47.997360 6508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:47.998381 6503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:47.999753 6510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 433a532ae71a48c8a2f56fa143752d2a. Latest consensus state: current_term: 1 leader_uuid: "433a532ae71a48c8a2f56fa143752d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } } }
I20260620 12:16:47.999987 6510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:47.999715 6509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "433a532ae71a48c8a2f56fa143752d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } } }
I20260620 12:16:48.000190 6509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:48.000691 6517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:16:48.004289 6517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:16:48.010763 6517 catalog_manager.cc:1347] Generated new cluster ID: ca306f4718a74f1db7d43a85fdd34781
I20260620 12:16:48.010890 6517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:16:48.029412 6517 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:16:48.030542 6517 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:16:48.040688 6517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Generated new TSK 0
I20260620 12:16:48.041669 6517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:16:48.077821 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45297
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:48.215929 6529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:48.216264 6529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:48.216327 6529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:48.220856 6529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:48.220976 6529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:48.221099 6529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:16:48.228367 6529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:48.229760 6529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:48.230967 6529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:48.244318 6534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:48.244612 6529 server_base.cc:1034] running on GCE node
W20260620 12:16:48.244426 6537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:48.244426 6535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:48.245445 6529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:48.246239 6529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:48.247450 6529 hybrid_clock.cc:648] HybridClock initialized: now 1781957808247420 us; error 45 us; skew 500 ppm
Jun 20 12:16:48 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957808, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:48.250909 6529 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:16:48.252046 6529 webserver.cc:460] Webserver started at http://127.24.141.1:39533/ using document root <none> and password file <none>
I20260620 12:16:48.252718 6529 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:48.252794 6529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:48.253005 6529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:48.255434 6529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "d3adb4e587b44462a247255f73cbd38e"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "d9c93e610a47546f60d19027ac6cad4e"
server_key_iv: "4f48dba0cefb6454c498c3fa5f03c7c5"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.256125 6529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "d3adb4e587b44462a247255f73cbd38e"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "d9c93e610a47546f60d19027ac6cad4e"
server_key_iv: "4f48dba0cefb6454c498c3fa5f03c7c5"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.262207 6529 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 12:16:48.266204 6544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.268690 6529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:16:48.268893 6529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "d3adb4e587b44462a247255f73cbd38e"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "d9c93e610a47546f60d19027ac6cad4e"
server_key_iv: "4f48dba0cefb6454c498c3fa5f03c7c5"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.269037 6529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:48.288475 6529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:48.292709 6529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:48.293038 6529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:48.294171 6529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:48.295259 6529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:48.295326 6529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.295377 6529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:48.295403 6529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.307134 6529 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:34189
I20260620 12:16:48.307209 6657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:34189 every 8 connection(s)
I20260620 12:16:48.308350 6529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:16:48.308606 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6529
I20260620 12:16:48.308755 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:16:48.309132 25140 external_mini_cluster.cc:1428] Setting key f3e3144b206d7e454afbba0d86468764
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
Jun 20 12:16:48 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957808, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:48.331099 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.311300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:41919 (local address 127.24.141.62:45297)
0620 12:16:48.311470 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.311474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.312966 (+ 1492us) server_negotiation.cc:407] Connection header received
0620 12:16:48.315169 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.315174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.315256 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.315339 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:48.317281 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.317906 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.319373 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.319644 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.319751 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.325169 (+ 5418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:48.325174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:48.325178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:48.325227 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:48.327189 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.328360 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.328364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.328365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.328532 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.329256 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.329259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.329261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.329584 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:48.329687 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.330731 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.330913 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:16:48.333158 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.311930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:41919)
0620 12:16:48.312704 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.312812 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.314838 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.315639 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.315651 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.316213 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:48.316988 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.317016 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.318070 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.318073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.319210 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.319223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.319841 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.319845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.320065 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.320799 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:48.320821 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:48.324812 (+ 3991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:48.327386 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:48.327396 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:48.327424 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:48.328100 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:48.328642 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:48.328645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:48.328648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:48.329000 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:48.329760 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:48.329771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:48.330341 (+ 570us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.330641 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.331135 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":167,"threads_started":1}
I20260620 12:16:48.335084 6658 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45297
I20260620 12:16:48.335443 6658 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:48.336294 6658 heartbeater.cc:511] Master 127.24.141.62:45297 requested a full tablet report, sending...
I20260620 12:16:48.338815 6467 ts_manager.cc:194] Registered new tserver with Master: d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189)
I20260620 12:16:48.340448 6467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:41919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:16:48.345014 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45297
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:48.486443 6663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:48.486773 6663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:48.486819 6663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:48.491077 6663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:48.491168 6663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:48.491233 6663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:16:48.499461 6663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:48.500783 6663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:48.502063 6663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:16:48.516834 6663 server_base.cc:1034] running on GCE node
W20260620 12:16:48.516784 6668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:48.516857 6669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:48.517050 6671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:48.517634 6663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:48.518458 6663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:48.519680 6663 hybrid_clock.cc:648] HybridClock initialized: now 1781957808519645 us; error 54 us; skew 500 ppm
Jun 20 12:16:48 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957808, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:48.523329 6663 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:16:48.524519 6663 webserver.cc:460] Webserver started at http://127.24.141.2:42803/ using document root <none> and password file <none>
I20260620 12:16:48.525202 6663 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:48.525282 6663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:48.525561 6663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:48.527989 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "5712a734f32c071f6ec4a8f4730ef307"
server_key_iv: "5bac9da23281cd2e427e1d899f791dbd"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.528613 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "5712a734f32c071f6ec4a8f4730ef307"
server_key_iv: "5bac9da23281cd2e427e1d899f791dbd"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.533923 6663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:16:48.537725 6678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.539765 6663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:16:48.539979 6663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "5712a734f32c071f6ec4a8f4730ef307"
server_key_iv: "5bac9da23281cd2e427e1d899f791dbd"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.540163 6663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:48.574944 6663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:48.579280 6663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:48.579696 6663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:48.580514 6663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:48.581624 6663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:48.581686 6663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.581763 6663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:48.581791 6663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.592283 6663 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:38829
I20260620 12:16:48.592362 6791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:38829 every 8 connection(s)
I20260620 12:16:48.593865 6663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:16:48.595731 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6663
I20260620 12:16:48.595991 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:16:48.596647 25140 external_mini_cluster.cc:1428] Setting key 7d388d1ed9062d3544ee82de5924d92d
Jun 20 12:16:48 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957808, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:16:48.614005 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.596878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:44157 (local address 127.24.141.62:45297)
0620 12:16:48.597141 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.597147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.598770 (+ 1623us) server_negotiation.cc:407] Connection header received
0620 12:16:48.600944 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.600950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.601094 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.601254 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:48.602789 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.603470 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.605106 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.605353 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.605457 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.609054 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:48.609092 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:48.609096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:48.609186 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:48.610740 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.611758 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.611762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.611764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.611886 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.612402 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.612405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.612414 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.612695 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:48.612798 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.613640 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.613808 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 12:16:48.615442 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.597600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:44157)
0620 12:16:48.598545 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.598644 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.600568 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.601509 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.601519 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.601956 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:48.602552 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.602568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.603711 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.603716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.604913 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.604929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.605548 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.605552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.605777 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.606500 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:48.606520 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:48.608773 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:48.610948 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:48.610960 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:48.610983 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:48.611602 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:48.612057 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:48.612060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:48.612062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:48.612290 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:48.612821 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:48.612829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:48.613270 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.613562 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.614043 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":262,"threads_started":1}
I20260620 12:16:48.617311 6792 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45297
I20260620 12:16:48.617717 6792 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:48.618531 6792 heartbeater.cc:511] Master 127.24.141.62:45297 requested a full tablet report, sending...
I20260620 12:16:48.620218 6467 ts_manager.cc:194] Registered new tserver with Master: 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829)
I20260620 12:16:48.620975 6467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:44157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:16:48.628885 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45297
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:48.765698 6797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:48.766038 6797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:48.766084 6797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:48.770879 6797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:48.770990 6797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:48.771059 6797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:16:48.777918 6797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:48.779234 6797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:48.780421 6797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:48.792236 6805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:48.792249 6802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:48.792335 6797 server_base.cc:1034] running on GCE node
W20260620 12:16:48.792249 6803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:48.793215 6797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:48.794106 6797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:48.795311 6797 hybrid_clock.cc:648] HybridClock initialized: now 1781957808795294 us; error 45 us; skew 500 ppm
Jun 20 12:16:48 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957808, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:48.799098 6797 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:16:48.800371 6797 webserver.cc:460] Webserver started at http://127.24.141.3:34499/ using document root <none> and password file <none>
I20260620 12:16:48.801023 6797 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:48.801141 6797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:48.801357 6797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:16:48.803941 6797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "f67939c8a19f4d09a9a8481534055c08"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "c389175625f298068311018af12fad0f"
server_key_iv: "2a3f3ef2f0eefceb4b1b183810f54a45"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.804870 6797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "f67939c8a19f4d09a9a8481534055c08"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "c389175625f298068311018af12fad0f"
server_key_iv: "2a3f3ef2f0eefceb4b1b183810f54a45"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.810657 6797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:16:48.814414 6812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.816265 6797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:16:48.816450 6797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "f67939c8a19f4d09a9a8481534055c08"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "c389175625f298068311018af12fad0f"
server_key_iv: "2a3f3ef2f0eefceb4b1b183810f54a45"
server_key_version: "encryptionkey@0"
I20260620 12:16:48.816591 6797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:48.842209 6797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:48.846117 6797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:48.846307 6797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:48.846935 6797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:48.848119 6797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:48.848186 6797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.848248 6797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:48.848285 6797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:48.858915 6797 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:43563
I20260620 12:16:48.858949 6925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:43563 every 8 connection(s)
I20260620 12:16:48.859985 6797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:16:48.869323 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6797
I20260620 12:16:48.869498 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:16:48.869860 25140 external_mini_cluster.cc:1428] Setting key e9a33d7c0fd8b22ca93b2ba0db058725
Jun 20 12:16:48 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957808, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:48.879683 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.862699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:37259 (local address 127.24.141.62:45297)
0620 12:16:48.862976 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.862982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.864131 (+ 1149us) server_negotiation.cc:407] Connection header received
0620 12:16:48.866022 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.866027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.866097 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.866205 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:48.868109 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.868736 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.870899 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.871294 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.871433 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.875405 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:48.875409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:48.875411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:48.875445 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:48.877100 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.877778 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.877781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.877782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.877871 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.878252 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.878256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.878258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.878488 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:48.878581 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.879227 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.879404 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260620 12:16:48.880664 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.863108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:37259)
0620 12:16:48.863848 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.863924 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.865629 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.866442 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.866451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.866995 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:48.867787 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.867809 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.868952 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.868956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.870663 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.870683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.871542 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.871549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.871811 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.872674 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:48.872696 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:48.875115 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:48.877265 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:48.877272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:48.877292 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:48.877650 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:48.877965 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:48.877967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:48.877969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:48.878140 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:48.878599 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:48.878608 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:48.878943 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.879146 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.879656 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":161,"threads_started":1}
I20260620 12:16:48.882521 6926 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45297
I20260620 12:16:48.882961 6926 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:48.883881 6926 heartbeater.cc:511] Master 127.24.141.62:45297 requested a full tablet report, sending...
I20260620 12:16:48.885542 6467 ts_manager.cc:194] Registered new tserver with Master: f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563)
I20260620 12:16:48.886107 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:16:48.886379 6467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:37259
I20260620 12:16:48.900216 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.889339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.24.141.62:45297)
0620 12:16:48.889560 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.889565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.889931 (+ 366us) server_negotiation.cc:407] Connection header received
0620 12:16:48.890122 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.890126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.890192 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.890291 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:48.891303 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.892076 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.893521 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.893816 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.893968 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.895857 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:48.895861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:48.895864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:48.895908 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:48.897708 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.898496 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.898500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.898501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.898633 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:48.898998 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:48.899001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:48.899003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:48.899361 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:48.899488 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.899850 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.900021 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 12:16:48.904392 6467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:16:48.908339 6467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:16:48.928380 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.921764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.24.141.1:34189)
0620 12:16:48.922257 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.922265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.922292 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:16:48.922384 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.922390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.922631 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.922798 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:48.923314 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.924608 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.926558 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.927403 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.927549 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.927753 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.928059 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.928221 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":179,"threads_started":1}
I20260620 12:16:48.928331 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.921329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34189 (local address 127.0.0.1:32844)
0620 12:16:48.922083 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.922099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.922208 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.922975 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.922978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.922991 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:48.923160 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.923167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.924808 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.924811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.926392 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.926405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.927619 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.927623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.927758 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.927821 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.927966 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.928143 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":141,"thread_start_us":136,"threads_started":1}
I20260620 12:16:48.929265 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.920924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38829 (local address 127.0.0.1:34508)
0620 12:16:48.921618 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.921702 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.921915 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.922828 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.922835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.922863 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:48.923102 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.923112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.924762 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.924765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.926782 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.926804 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.928043 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.928047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.928147 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.928175 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.928226 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.928326 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":259,"thread_start_us":145,"threads_started":1}
I20260620 12:16:48.929684 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.922397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:43563 (local address 127.0.0.1:54612)
0620 12:16:48.923046 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.923060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.923176 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.923641 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.923645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.923662 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:48.923812 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.923819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.924876 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.924879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.926782 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.926804 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.929237 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.929241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.929333 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.929359 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.929440 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.929546 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":249,"thread_start_us":118,"threads_started":1}
I20260620 12:16:48.930171 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.922596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.24.141.3:43563)
0620 12:16:48.923076 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.923094 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.923135 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:16:48.923316 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.923322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.923509 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.923656 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:48.923925 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.924722 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.927858 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.928699 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.928982 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.929174 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.929864 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.930012 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":142,"threads_started":1}
I20260620 12:16:48.930387 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.921094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.24.141.2:38829)
0620 12:16:48.922258 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.922269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.922292 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:16:48.922384 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.922390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.922631 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.923254 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:48.923434 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.924608 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.927001 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.927878 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.929661 (+ 1783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.930042 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.930102 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.930218 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":175,"threads_started":1}
I20260620 12:16:48.932042 6592 tablet_service.cc:1467] Processing CreateTablet for tablet 4c369b0d36f1428bad75bbec6f90a588 (DEFAULT_TABLE table=test-table [id=a329f5ad94f4467a8f85ccd25a4d23fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:48.933517 6592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c369b0d36f1428bad75bbec6f90a588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:48.933928 6860 tablet_service.cc:1467] Processing CreateTablet for tablet 4c369b0d36f1428bad75bbec6f90a588 (DEFAULT_TABLE table=test-table [id=a329f5ad94f4467a8f85ccd25a4d23fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:48.933957 6726 tablet_service.cc:1467] Processing CreateTablet for tablet 4c369b0d36f1428bad75bbec6f90a588 (DEFAULT_TABLE table=test-table [id=a329f5ad94f4467a8f85ccd25a4d23fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:48.935250 6860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c369b0d36f1428bad75bbec6f90a588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:48.935300 6726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c369b0d36f1428bad75bbec6f90a588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:48.943545 6943 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Bootstrap starting.
I20260620 12:16:48.945658 6944 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Bootstrap starting.
I20260620 12:16:48.946748 6945 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Bootstrap starting.
I20260620 12:16:48.948057 6943 tablet_bootstrap.cc:654] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:48.948545 6944 tablet_bootstrap.cc:654] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:48.949307 6943 log.cc:824] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:48.949560 6944 log.cc:824] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:48.949748 6945 tablet_bootstrap.cc:654] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:48.951150 6945 log.cc:824] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:48.952212 6944 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: No bootstrap required, opened a new log
I20260620 12:16:48.952212 6943 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: No bootstrap required, opened a new log
I20260620 12:16:48.952499 6943 ts_tablet_manager.cc:1397] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 12:16:48.953327 6944 ts_tablet_manager.cc:1397] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:16:48.953657 6945 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: No bootstrap required, opened a new log
I20260620 12:16:48.953871 6945 ts_tablet_manager.cc:1397] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:16:48.956357 6943 raft_consensus.cc:348] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.956679 6943 raft_consensus.cc:374] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:48.956729 6943 raft_consensus.cc:729] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041eae0c0c9e4f89b58cdf2bc176ddb9, State: Initialized, Role: FOLLOWER
I20260620 12:16:48.957343 6943 consensus_queue.cc:260] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.958195 6792 heartbeater.cc:503] Master 127.24.141.62:45297 was elected leader, sending a full tablet report...
I20260620 12:16:48.958225 6945 raft_consensus.cc:348] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.958613 6945 raft_consensus.cc:374] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:48.958664 6945 raft_consensus.cc:729] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67939c8a19f4d09a9a8481534055c08, State: Initialized, Role: FOLLOWER
I20260620 12:16:48.958652 6943 ts_tablet_manager.cc:1428] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:16:48.958505 6944 raft_consensus.cc:348] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.958734 6944 raft_consensus.cc:374] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:48.958774 6944 raft_consensus.cc:729] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3adb4e587b44462a247255f73cbd38e, State: Initialized, Role: FOLLOWER
I20260620 12:16:48.959275 6945 consensus_queue.cc:260] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.959275 6944 consensus_queue.cc:260] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.960444 6658 heartbeater.cc:503] Master 127.24.141.62:45297 was elected leader, sending a full tablet report...
I20260620 12:16:48.960588 6945 ts_tablet_manager.cc:1428] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:16:48.960899 6944 ts_tablet_manager.cc:1428] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:16:48.961254 6926 heartbeater.cc:503] Master 127.24.141.62:45297 was elected leader, sending a full tablet report...
I20260620 12:16:48.973578 6950 raft_consensus.cc:482] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:16:48.973847 6950 raft_consensus.cc:504] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.975193 6950 leader_election.cc:288] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829), d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189)
I20260620 12:16:48.980005 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.975672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:59329 (local address 127.24.141.1:34189)
0620 12:16:48.975792 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.975796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.976004 (+ 208us) server_negotiation.cc:407] Connection header received
0620 12:16:48.976071 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.976075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.976127 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.976191 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:48.976552 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.977157 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.978912 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.979386 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.979516 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.979580 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.979778 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.979861 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:16:48.980005 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.975520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34189 (local address 127.24.141.3:59329)
0620 12:16:48.975776 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.975792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.975914 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.976303 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.976305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.976320 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:48.976425 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.976431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.977293 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.977297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.978738 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.978752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.979522 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.979526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.979650 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.979673 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.979744 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.979847 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1}
I20260620 12:16:48.980556 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.976151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:55935 (local address 127.24.141.2:38829)
0620 12:16:48.976299 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:48.976304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:48.976322 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:16:48.976396 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:48.976399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:48.976459 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:48.976549 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:48.977094 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.977675 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.979034 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.979734 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.979872 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.979935 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:48.980290 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 12:16:48.980395 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:16:48.980890 6612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c369b0d36f1428bad75bbec6f90a588" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3adb4e587b44462a247255f73cbd38e" is_pre_election: true
I20260620 12:16:48.981151 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:48.975557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38829 (local address 127.24.141.3:55935)
0620 12:16:48.976018 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:48.976033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:48.976109 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:48.976691 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:48.976695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:48.976721 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:48.976878 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.976886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.977829 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.977834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.978798 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:48.978811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:48.980031 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:48.980035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:48.980147 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:48.980173 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:16:48.980827 (+ 654us) client_negotiation.cc:240] Negotiation successful
0620 12:16:48.980934 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":55,"thread_start_us":175,"threads_started":1}
I20260620 12:16:48.981312 6612 raft_consensus.cc:2454] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 0.
I20260620 12:16:48.981913 6746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c369b0d36f1428bad75bbec6f90a588" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" is_pre_election: true
I20260620 12:16:48.982050 6813 leader_election.cc:302] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e, f67939c8a19f4d09a9a8481534055c08; no voters:
I20260620 12:16:48.982344 6746 raft_consensus.cc:2454] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 0.
I20260620 12:16:48.982424 6950 raft_consensus.cc:2781] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:16:48.982573 6950 raft_consensus.cc:482] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:16:48.982628 6950 raft_consensus.cc:3037] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:48.984086 6950 raft_consensus.cc:504] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.984522 6950 leader_election.cc:288] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 election: Requested vote from peers 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829), d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189)
I20260620 12:16:48.985033 6746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c369b0d36f1428bad75bbec6f90a588" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9"
I20260620 12:16:48.985096 6612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c369b0d36f1428bad75bbec6f90a588" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3adb4e587b44462a247255f73cbd38e"
I20260620 12:16:48.985255 6746 raft_consensus.cc:3037] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:48.985255 6612 raft_consensus.cc:3037] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:48.986639 6612 raft_consensus.cc:2454] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 1.
I20260620 12:16:48.986855 6746 raft_consensus.cc:2454] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 1.
I20260620 12:16:48.986996 6813 leader_election.cc:302] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e, f67939c8a19f4d09a9a8481534055c08; no voters:
I20260620 12:16:48.987424 6950 raft_consensus.cc:2781] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:16:48.987701 6950 raft_consensus.cc:686] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 1 LEADER]: Becoming Leader. State: Replica: f67939c8a19f4d09a9a8481534055c08, State: Running, Role: LEADER
I20260620 12:16:48.988097 6950 consensus_queue.cc:237] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:48.992048 6464 catalog_manager.cc:5495] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 reported cstate change: term changed from 0 to 1, leader changed from <none> to f67939c8a19f4d09a9a8481534055c08 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "f67939c8a19f4d09a9a8481534055c08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } health_report { overall_health: UNKNOWN } } }
I20260620 12:16:49.015368 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.011236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.24.141.3:43563)
0620 12:16:49.011387 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:49.011391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:49.011411 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:16:49.011546 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:49.011548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:49.011612 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:49.011719 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:16:49.011985 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.012500 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.013531 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.013783 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.013891 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:49.014285 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:16:49.014838 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:16:49.014948 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:49.015130 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 12:16:49.015208 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:16:49.024531 6612 raft_consensus.cc:1261] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Refusing update from remote peer f67939c8a19f4d09a9a8481534055c08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:49.024756 6746 raft_consensus.cc:1261] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Refusing update from remote peer f67939c8a19f4d09a9a8481534055c08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:49.025658 6953 consensus_queue.cc:1035] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:16:49.026033 6950 consensus_queue.cc:1035] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:16:49.039549 6958 mvcc.cc:205] Tried to move back new op lower bound from 7298899185754664960 to 7298899185618472960. Current Snapshot: MvccSnapshot[applied={T|T < 7298899185754664960}]
I20260620 12:16:49.039824 6957 mvcc.cc:205] Tried to move back new op lower bound from 7298899185754664960 to 7298899185618472960. Current Snapshot: MvccSnapshot[applied={T|T < 7298899185754664960}]
I20260620 12:16:49.050401 6464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:16:49.050613 6464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:16:49.054365 6464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Sending DeleteTablet for 3 replicas of tablet 4c369b0d36f1428bad75bbec6f90a588
I20260620 12:16:49.055759 6726 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c369b0d36f1428bad75bbec6f90a588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:49 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:34508
I20260620 12:16:49.055869 6860 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c369b0d36f1428bad75bbec6f90a588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:49 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:54612
I20260620 12:16:49.056407 6966 tablet_replica.cc:330] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: stopping tablet replica
I20260620 12:16:49.056497 6592 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c369b0d36f1428bad75bbec6f90a588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:49 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:32844
I20260620 12:16:49.056860 6967 tablet_replica.cc:330] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: stopping tablet replica
I20260620 12:16:49.056965 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6529
I20260620 12:16:49.057349 6967 raft_consensus.cc:2229] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:16:49.058015 6967 raft_consensus.cc:2258] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:49.059077 6966 raft_consensus.cc:2229] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:16:49.059659 6966 raft_consensus.cc:2258] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:49.060796 6967 ts_tablet_manager.cc:1905] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:16:49.063345 6966 ts_tablet_manager.cc:1905] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:16:49.064620 6967 ts_tablet_manager.cc:1918] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:16:49.064798 6967 log.cc:1196] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/4c369b0d36f1428bad75bbec6f90a588
I20260620 12:16:49.065210 6967 ts_tablet_manager.cc:1939] T 4c369b0d36f1428bad75bbec6f90a588 P f67939c8a19f4d09a9a8481534055c08: Deleting consensus metadata
I20260620 12:16:49.067160 6452 catalog_manager.cc:4841] TS f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563): tablet 4c369b0d36f1428bad75bbec6f90a588 (table test-table [id=a329f5ad94f4467a8f85ccd25a4d23fb]) successfully deleted
I20260620 12:16:49.067253 6966 ts_tablet_manager.cc:1918] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:16:49.067586 6966 log.cc:1196] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/4c369b0d36f1428bad75bbec6f90a588
I20260620 12:16:49.067941 6966 ts_tablet_manager.cc:1939] T 4c369b0d36f1428bad75bbec6f90a588 P 041eae0c0c9e4f89b58cdf2bc176ddb9: Deleting consensus metadata
W20260620 12:16:49.068127 6451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:34189 (error 108)
I20260620 12:16:49.069392 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6663
I20260620 12:16:49.069530 6452 catalog_manager.cc:4841] TS 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829): tablet 4c369b0d36f1428bad75bbec6f90a588 (table test-table [id=a329f5ad94f4467a8f85ccd25a4d23fb]) successfully deleted
I20260620 12:16:49.070780 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.069160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34189 (local address 127.0.0.1:32846)
0620 12:16:49.069393 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:49.069492 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260620 12:16:49.070997 6451 catalog_manager.cc:4584] TS d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c369b0d36f1428bad75bbec6f90a588: Network error: Client connection negotiation failed: client connection to 127.24.141.1:34189: connect: Connection refused (error 111)
I20260620 12:16:49.078002 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6797
I20260620 12:16:49.085523 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6435
I20260620 12:16:49.093554 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45297
--webserver_interface=127.24.141.62
--webserver_port=42385
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:49.223006 6970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:49.223345 6970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:49.223395 6970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:49.229528 6970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:49.229643 6970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:49.229665 6970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:49.229682 6970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:49.229701 6970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:49.236438 6970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:49.237797 6970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:49.238925 6970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:49.255805 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:49.255805 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:49.255797 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:49.256316 6970 server_base.cc:1034] running on GCE node
I20260620 12:16:49.257148 6970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:49.258848 6970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:49.260181 6970 hybrid_clock.cc:648] HybridClock initialized: now 1781957809260123 us; error 84 us; skew 500 ppm
Jun 20 12:16:49 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957809, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:49.264518 6970 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:16:49.265890 6970 webserver.cc:460] Webserver started at http://127.24.141.62:42385/ using document root <none> and password file <none>
I20260620 12:16:49.266585 6970 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:49.266682 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:49.271785 6970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:16:49.274933 6986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:49.276778 6970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 12:16:49.277014 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "433a532ae71a48c8a2f56fa143752d2a"
format_stamp: "Formatted at 2026-06-20 12:16:47 on dist-test-slave-rh5v"
server_key: "b1ba99d1daa9441d94738e5f843e6c1e"
server_key_iv: "e2b89f221548afedb73f1e5040274618"
server_key_version: "encryptionkey@0"
I20260620 12:16:49.277810 6970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:49.320629 6970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:49.324795 6970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:49.325047 6970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:49.334640 6970 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:45297
I20260620 12:16:49.334973 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:45297 every 8 connection(s)
I20260620 12:16:49.336185 6970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:16:49.341847 7039 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:16:49.343982 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 6970
I20260620 12:16:49.344030 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Bootstrap starting.
I20260620 12:16:49.344581 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:34189
--local_ip_for_outbound_sockets=127.24.141.1
--tserver_master_addrs=127.24.141.62:45297
--webserver_port=39533
--webserver_interface=127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:16:49.363358 7039 log.cc:824] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:49.373596 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:16:49.374025 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Bootstrap complete.
I20260620 12:16:49.378476 7039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:49.378994 7039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a532ae71a48c8a2f56fa143752d2a, State: Initialized, Role: FOLLOWER
I20260620 12:16:49.379571 7039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:49.379722 7039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:16:49.379820 7039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:16:49.379990 7039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:16:49.382459 7039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:49.382886 7039 leader_election.cc:302] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 433a532ae71a48c8a2f56fa143752d2a; no voters:
I20260620 12:16:49.383242 7039 leader_election.cc:288] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:16:49.383395 7043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:16:49.383618 7043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 LEADER]: Becoming Leader. State: Replica: 433a532ae71a48c8a2f56fa143752d2a, State: Running, Role: LEADER
I20260620 12:16:49.383980 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:49.384755 7039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:49.385830 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 433a532ae71a48c8a2f56fa143752d2a. Latest consensus state: current_term: 2 leader_uuid: "433a532ae71a48c8a2f56fa143752d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } } }
I20260620 12:16:49.385952 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "433a532ae71a48c8a2f56fa143752d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } } }
I20260620 12:16:49.386122 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:49.386063 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:49.386924 7052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:16:49.393368 7052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:16:49.394249 7052 catalog_manager.cc:1259] Loaded cluster ID: ca306f4718a74f1db7d43a85fdd34781
I20260620 12:16:49.394336 7052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:16:49.396490 7052 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 12:16:49.400493 7052 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 12:16:49.406586 7052 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:16:49.407497 7052 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Loaded TSK: 0
I20260620 12:16:49.408773 7052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:16:49.505266 7041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:49.505589 7041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:49.505650 7041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:49.510577 7041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:49.510694 7041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:49.510793 7041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:16:49.517701 7041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:34189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=39533
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:49.518968 7041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:49.520272 7041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:49.533368 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:49.533433 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:49.533849 7041 server_base.cc:1034] running on GCE node
W20260620 12:16:49.533408 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:49.534614 7041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:49.535821 7041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:49.537262 7041 hybrid_clock.cc:648] HybridClock initialized: now 1781957809537207 us; error 82 us; skew 500 ppm
Jun 20 12:16:49 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957809, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:49.540524 7041 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:16:49.541810 7041 webserver.cc:460] Webserver started at http://127.24.141.1:39533/ using document root <none> and password file <none>
I20260620 12:16:49.542428 7041 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:49.542517 7041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:49.547562 7041 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:16:49.550580 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:49.552120 7041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:16:49.552268 7041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "d3adb4e587b44462a247255f73cbd38e"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "d9c93e610a47546f60d19027ac6cad4e"
server_key_iv: "4f48dba0cefb6454c498c3fa5f03c7c5"
server_key_version: "encryptionkey@0"
I20260620 12:16:49.552923 7041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:49.569550 7041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:49.573205 7041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:49.573422 7041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:49.574123 7041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:49.575788 7083 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:16:49.581809 7041 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:16:49.581956 7041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 12:16:49.582051 7041 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:16:49.584362 7041 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:16:49.584439 7041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 12:16:49.584939 7083 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Bootstrap starting.
I20260620 12:16:49.596127 7041 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:34189
I20260620 12:16:49.596159 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:34189 every 8 connection(s)
I20260620 12:16:49.597658 7041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:16:49.605196 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 7041
I20260620 12:16:49.605795 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:38829
--local_ip_for_outbound_sockets=127.24.141.2
--tserver_master_addrs=127.24.141.62:45297
--webserver_port=42803
--webserver_interface=127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:16:49.613351 7083 log.cc:824] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:16:49 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957809, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:49.621424 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.600329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39331 (local address 127.24.141.62:45297)
0620 12:16:49.601237 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:49.601266 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:49.601341 (+ 75us) server_negotiation.cc:407] Connection header received
0620 12:16:49.602759 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:49.602783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:49.603434 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:49.604159 (+ 725us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:49.606786 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.608925 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.611402 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.612029 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.612351 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:49.615539 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:49.615547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:49.615563 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:49.615601 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:49.617479 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:49.618182 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:49.618188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:49.618192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:49.618334 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:49.618704 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:49.618706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:49.618707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:49.619144 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:49.619275 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:49.619985 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 12:16:49.620324 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":216,"threads_started":1}
I20260620 12:16:49.621449 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.600206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:39331)
0620 12:16:49.600869 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:49.600936 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:49.601974 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:49.604211 (+ 2237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:49.604221 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:49.605454 (+ 1233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:49.606406 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:49.606437 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.609291 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.609297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:49.611135 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:49.611172 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.612314 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.612327 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:49.612563 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:49.613270 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:49.613290 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:49.615204 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:49.617653 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:49.617661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:49.617679 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:49.618034 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:49.618439 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:49.618442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:49.618444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:49.618596 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:49.619271 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:49.619278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:49.619549 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 12:16:49.619724 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 12:16:49.620274 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":188,"threads_started":1}
I20260620 12:16:49.624938 7191 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45297
I20260620 12:16:49.625329 7191 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:49.626011 7191 heartbeater.cc:511] Master 127.24.141.62:45297 requested a full tablet report, sending...
I20260620 12:16:49.627548 7083 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:16:49.628120 7083 tablet_bootstrap.cc:492] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Bootstrap complete.
I20260620 12:16:49.629630 7003 ts_manager.cc:194] Registered new tserver with Master: d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189)
I20260620 12:16:49.629737 7083 ts_tablet_manager.cc:1397] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Time spent bootstrapping tablet: real 0.045s user 0.031s sys 0.010s
I20260620 12:16:49.634083 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:39331
I20260620 12:16:49.633944 7083 raft_consensus.cc:348] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:49.634717 7083 raft_consensus.cc:729] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3adb4e587b44462a247255f73cbd38e, State: Initialized, Role: FOLLOWER
I20260620 12:16:49.635456 7083 consensus_queue.cc:260] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:49.636880 7083 ts_tablet_manager.cc:1428] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260620 12:16:49.637609 7191 heartbeater.cc:503] Master 127.24.141.62:45297 was elected leader, sending a full tablet report...
I20260620 12:16:49.641943 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.634564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.24.141.1:34189)
0620 12:16:49.635733 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:49.635746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:49.636414 (+ 668us) server_negotiation.cc:407] Connection header received
0620 12:16:49.636489 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:49.636495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:49.636709 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:49.636895 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:49.637477 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.638453 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.640438 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.641052 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.641235 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:49.641412 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:49.641641 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 12:16:49.641762 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":115,"threads_started":1}
I20260620 12:16:49.642081 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.633512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34189 (local address 127.0.0.1:32854)
0620 12:16:49.636043 (+ 2531us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:49.636098 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:49.636271 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:49.637154 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:49.637161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:49.637191 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:49.637335 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:49.637344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.638645 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.638651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:49.640217 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:49.640241 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.641250 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.641254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:49.641433 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:49.641479 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 12:16:49.641596 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 12:16:49.641776 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2335,"thread_start_us":143,"threads_started":1}
I20260620 12:16:49.643584 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c369b0d36f1428bad75bbec6f90a588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:49 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:32854
I20260620 12:16:49.645102 7200 tablet_replica.cc:330] stopping tablet replica
I20260620 12:16:49.645516 7200 raft_consensus.cc:2229] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:16:49.645833 7200 raft_consensus.cc:2258] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:49.646999 7200 ts_tablet_manager.cc:1905] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:16:49.651546 7200 ts_tablet_manager.cc:1918] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:16:49.651683 7200 log.cc:1196] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/4c369b0d36f1428bad75bbec6f90a588
I20260620 12:16:49.651929 7200 ts_tablet_manager.cc:1939] T 4c369b0d36f1428bad75bbec6f90a588 P d3adb4e587b44462a247255f73cbd38e: Deleting consensus metadata
I20260620 12:16:49.654176 6987 catalog_manager.cc:4841] TS d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189): tablet 4c369b0d36f1428bad75bbec6f90a588 (table test-table [id=a329f5ad94f4467a8f85ccd25a4d23fb]) successfully deleted
W20260620 12:16:49.758721 7195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:49.759146 7195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:49.759199 7195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:49.764003 7195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:49.764132 7195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:49.764204 7195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:16:49.771035 7195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:38829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=42803
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:49.772715 7195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:49.774360 7195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:49.787712 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:49.787712 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:49.787739 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:49.788028 7195 server_base.cc:1034] running on GCE node
I20260620 12:16:49.788647 7195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:49.789631 7195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:49.790824 7195 hybrid_clock.cc:648] HybridClock initialized: now 1781957809790807 us; error 48 us; skew 500 ppm
Jun 20 12:16:49 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957809, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:49.794903 7195 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:16:49.796164 7195 webserver.cc:460] Webserver started at http://127.24.141.2:42803/ using document root <none> and password file <none>
I20260620 12:16:49.796818 7195 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:49.796906 7195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:49.802171 7195 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:16:49.805706 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:49.807806 7195 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:16:49.808005 7195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "5712a734f32c071f6ec4a8f4730ef307"
server_key_iv: "5bac9da23281cd2e427e1d899f791dbd"
server_key_version: "encryptionkey@0"
I20260620 12:16:49.808813 7195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:49.842483 7195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:49.847064 7195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:49.847332 7195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:49.848591 7195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:49.850219 7195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:49.850318 7195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:49.850374 7195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:49.850389 7195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:49.863353 7195 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:38829
I20260620 12:16:49.863454 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:38829 every 8 connection(s)
I20260620 12:16:49.864688 7195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:16:49.867631 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 7195
I20260620 12:16:49.868223 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:43563
--local_ip_for_outbound_sockets=127.24.141.3
--tserver_master_addrs=127.24.141.62:45297
--webserver_port=34499
--webserver_interface=127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:16:49 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957809, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:49.888439 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.867483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:55501 (local address 127.24.141.62:45297)
0620 12:16:49.867704 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:49.867710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:49.869356 (+ 1646us) server_negotiation.cc:407] Connection header received
0620 12:16:49.871339 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:49.871344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:49.871434 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:49.871578 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:49.873722 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.874611 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.876765 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.877277 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.877468 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:49.881789 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:49.881794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:49.881798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:49.881852 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:49.884504 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:49.885659 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:49.885664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:49.885666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:49.885863 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:49.886516 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:49.886521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:49.886523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:49.886862 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:49.886984 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:49.887959 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 12:16:49.888206 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 12:16:49.889825 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:49.868102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:55501)
0620 12:16:49.868915 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:49.869149 (+ 234us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:49.870959 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:49.871923 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:49.871935 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:49.872523 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:49.873416 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:49.873446 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.874904 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.874909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:49.876488 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:49.876513 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:49.877605 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:49.877610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:49.877887 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:49.878912 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:49.878939 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:49.881294 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:49.884796 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:49.884807 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:49.884832 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:49.885469 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:49.886046 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:49.886050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:49.886052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:49.886327 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:49.887005 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:49.887014 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:49.887473 (+ 459us) client_negotiation.cc:769] Sending connection context
0620 12:16:49.887856 (+ 383us) client_negotiation.cc:240] Negotiation successful
0620 12:16:49.888373 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":236,"threads_started":1}
I20260620 12:16:49.891692 7329 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45297
I20260620 12:16:49.892027 7329 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:49.892741 7329 heartbeater.cc:511] Master 127.24.141.62:45297 requested a full tablet report, sending...
I20260620 12:16:49.894295 7003 ts_manager.cc:194] Registered new tserver with Master: 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829)
I20260620 12:16:49.895035 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:55501
W20260620 12:16:50.012852 7332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:50.013271 7332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:50.013357 7332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:50.019485 7332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:50.019594 7332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:50.019685 7332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:16:50.026918 7332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:43563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=34499
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:50.028209 7332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:50.029459 7332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:50.043200 7340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:50.043205 7338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:50.043205 7337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:50.043744 7332 server_base.cc:1034] running on GCE node
I20260620 12:16:50.044394 7332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:50.045416 7332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:50.046774 7332 hybrid_clock.cc:648] HybridClock initialized: now 1781957810046718 us; error 83 us; skew 500 ppm
Jun 20 12:16:50 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957810, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:50.050717 7332 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:16:50.052080 7332 webserver.cc:460] Webserver started at http://127.24.141.3:34499/ using document root <none> and password file <none>
I20260620 12:16:50.052852 7332 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:50.052965 7332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:50.058678 7332 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:16:50.062137 7347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:50.064599 7332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:16:50.064867 7332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "f67939c8a19f4d09a9a8481534055c08"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "c389175625f298068311018af12fad0f"
server_key_iv: "2a3f3ef2f0eefceb4b1b183810f54a45"
server_key_version: "encryptionkey@0"
I20260620 12:16:50.066131 7332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:50.095553 7332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:50.099718 7332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:50.099967 7332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:50.100760 7332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:50.101953 7332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:50.102011 7332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:50.102075 7332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:50.102103 7332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:50.114207 7332 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:43563
I20260620 12:16:50.114228 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:43563 every 8 connection(s)
I20260620 12:16:50.115696 7332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:16:50.118409 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 7332
Jun 20 12:16:50 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957810, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:16:50.130347 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.119312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.24.141.62:45297)
0620 12:16:50.119638 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.119643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.119997 (+ 354us) server_negotiation.cc:407] Connection header received
0620 12:16:50.120156 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.120159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.120230 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.120356 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:50.121311 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.122154 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.123295 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.123656 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.123791 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.125627 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:50.125631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:50.125634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:50.125680 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:50.127732 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:50.128406 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:50.128410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:50.128413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:50.128534 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:50.128953 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:50.128956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:50.128957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:50.129272 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:50.129612 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.129778 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.129959 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260620 12:16:50.134887 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.118159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:47095 (local address 127.24.141.62:45297)
0620 12:16:50.118334 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.118339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.119733 (+ 1394us) server_negotiation.cc:407] Connection header received
0620 12:16:50.121881 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.121885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.121949 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.122042 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:50.123817 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.124443 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.126026 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.126416 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.126538 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.130163 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:50.130167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:50.130169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:50.130203 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:50.132017 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:50.132905 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:50.132910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:50.132911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:50.133026 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:50.133492 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:50.133495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:50.133496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:50.133721 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:50.133813 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.134581 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.134725 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:16:50.136237 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.118704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:47095)
0620 12:16:50.119506 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.119592 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:50.121463 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:50.122345 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:50.122355 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:50.122857 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:50.123646 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.123667 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.124616 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.124619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.125826 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.125837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.126663 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.126668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.126923 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.127648 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:50.127679 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:50.129909 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:50.132186 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:50.132196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:50.132225 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:50.132749 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:50.133159 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:50.133162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:50.133164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:50.133369 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:50.133863 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:50.133871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:50.134258 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 12:16:50.134491 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 12:16:50.134948 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":171,"threads_started":1}
I20260620 12:16:50.138015 7461 heartbeater.cc:347] Connected to a master server at 127.24.141.62:45297
I20260620 12:16:50.138360 7461 heartbeater.cc:464] Registering TS with master...
I20260620 12:16:50.139048 7461 heartbeater.cc:511] Master 127.24.141.62:45297 requested a full tablet report, sending...
I20260620 12:16:50.140563 7003 ts_manager.cc:194] Registered new tserver with Master: f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563)
I20260620 12:16:50.141393 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:47095
I20260620 12:16:50.146376 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:16:50.152168 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.147268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.24.141.62:45297)
0620 12:16:50.147502 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.147508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.147770 (+ 262us) server_negotiation.cc:407] Connection header received
0620 12:16:50.148048 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.148051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.148207 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.148325 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:16:50.148636 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.149315 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.150292 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.150524 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.150622 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.151017 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:16:50.151639 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:16:50.151771 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.151841 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.151958 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 12:16:50.153251 7003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:16:50.157526 7003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:16:50.168464 7121 tablet_service.cc:1467] Processing CreateTablet for tablet eb857835183a424ca7af43d61bba722e (DEFAULT_TABLE table=test-table [id=7a316a9bcea5437693ecb97bc7e00f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:50.169567 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb857835183a424ca7af43d61bba722e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:50.174129 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.167458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.24.141.3:43563)
0620 12:16:50.167989 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.168008 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.168224 (+ 216us) server_negotiation.cc:407] Connection header received
0620 12:16:50.168527 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.168537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.168885 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.169113 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:50.169524 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.170708 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.172189 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.173001 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.173128 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.173305 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.173754 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.173891 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":144,"threads_started":1}
I20260620 12:16:50.174278 7474 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Bootstrap starting.
I20260620 12:16:50.174635 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.167201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:43563 (local address 127.0.0.1:54620)
0620 12:16:50.168084 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.168103 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:50.168335 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:50.169166 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:50.169169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:50.169189 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:50.169331 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.169349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.170923 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.170927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.172024 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.172038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.173454 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.173457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.173568 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.173595 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:16:50.174391 (+ 796us) client_negotiation.cc:240] Negotiation successful
0620 12:16:50.174499 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260620 12:16:50.175978 7474 tablet_bootstrap.cc:654] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:50.176401 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.167655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38829 (local address 127.0.0.1:34518)
0620 12:16:50.168217 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.168235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:50.168379 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:50.169386 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:50.169389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:50.169405 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:50.169482 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.169487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.171546 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.171550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.173294 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.173313 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.175744 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.175749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.175988 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.176021 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 12:16:50.176136 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 12:16:50.176246 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":137,"threads_started":1}
I20260620 12:16:50.177176 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.167913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.24.141.2:38829)
0620 12:16:50.168617 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.168626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.168650 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:16:50.168734 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.168740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.169006 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.169211 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:50.170109 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.171375 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.173537 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.175515 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.175696 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.175909 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.176783 (+ 874us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.176952 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":184,"threads_started":1}
I20260620 12:16:50.177599 7474 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: No bootstrap required, opened a new log
I20260620 12:16:50.177734 7474 ts_tablet_manager.cc:1397] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 12:16:50.178258 7474 raft_consensus.cc:348] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.178395 7474 raft_consensus.cc:374] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:50.178436 7474 raft_consensus.cc:729] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3adb4e587b44462a247255f73cbd38e, State: Initialized, Role: FOLLOWER
I20260620 12:16:50.178588 7474 consensus_queue.cc:260] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.178977 7474 ts_tablet_manager.cc:1428] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:16:50.178534 7395 tablet_service.cc:1467] Processing CreateTablet for tablet eb857835183a424ca7af43d61bba722e (DEFAULT_TABLE table=test-table [id=7a316a9bcea5437693ecb97bc7e00f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:50.180420 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb857835183a424ca7af43d61bba722e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:50.181908 7263 tablet_service.cc:1467] Processing CreateTablet for tablet eb857835183a424ca7af43d61bba722e (DEFAULT_TABLE table=test-table [id=7a316a9bcea5437693ecb97bc7e00f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:16:50.183696 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb857835183a424ca7af43d61bba722e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:16:50.191296 7477 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Bootstrap starting.
I20260620 12:16:50.194131 7478 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Bootstrap starting.
I20260620 12:16:50.195359 7477 tablet_bootstrap.cc:654] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:50.197113 7477 log.cc:824] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:50.197656 7478 tablet_bootstrap.cc:654] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Neither blocks nor log segments found. Creating new log.
I20260620 12:16:50.199034 7478 log.cc:824] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:50.200536 7477 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: No bootstrap required, opened a new log
I20260620 12:16:50.200862 7477 ts_tablet_manager.cc:1397] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 12:16:50.201570 7478 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: No bootstrap required, opened a new log
I20260620 12:16:50.201822 7478 ts_tablet_manager.cc:1397] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260620 12:16:50.205150 7477 raft_consensus.cc:348] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.205538 7477 raft_consensus.cc:374] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:50.205602 7477 raft_consensus.cc:729] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67939c8a19f4d09a9a8481534055c08, State: Initialized, Role: FOLLOWER
I20260620 12:16:50.205822 7478 raft_consensus.cc:348] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.206152 7478 raft_consensus.cc:374] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:16:50.206212 7478 raft_consensus.cc:729] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041eae0c0c9e4f89b58cdf2bc176ddb9, State: Initialized, Role: FOLLOWER
I20260620 12:16:50.206198 7477 consensus_queue.cc:260] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.206751 7478 consensus_queue.cc:260] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.207270 7461 heartbeater.cc:503] Master 127.24.141.62:45297 was elected leader, sending a full tablet report...
I20260620 12:16:50.207641 7329 heartbeater.cc:503] Master 127.24.141.62:45297 was elected leader, sending a full tablet report...
I20260620 12:16:50.207765 7477 ts_tablet_manager.cc:1428] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 12:16:50.208155 7478 ts_tablet_manager.cc:1428] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:16:50.302079 7481 raft_consensus.cc:482] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:16:50.302338 7481 raft_consensus.cc:504] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.303649 7481 leader_election.cc:288] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829), d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189)
I20260620 12:16:50.308554 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.303974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38829 (local address 127.24.141.3:53849)
0620 12:16:50.304369 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.304391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:50.304531 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:50.304842 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:50.304846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:50.304862 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:50.305007 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.305014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.306024 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.306028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.307051 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.307066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.307994 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.307998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.308129 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.308160 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:16:50.308265 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 12:16:50.308372 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260620 12:16:50.308544 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.304110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:53849 (local address 127.24.141.2:38829)
0620 12:16:50.304325 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.304331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.304558 (+ 227us) server_negotiation.cc:407] Connection header received
0620 12:16:50.304649 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.304652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.304725 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.304806 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:50.305168 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.305850 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.307219 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.307831 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.307958 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.308018 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.308303 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.308396 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 12:16:50.309170 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.304105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34189 (local address 127.24.141.3:46193)
0620 12:16:50.304595 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.304610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:50.304776 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:50.305155 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:50.305159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:50.305186 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:16:50.305341 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.305348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.306531 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.306535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.307592 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:50.307605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.308653 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.308657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:50.308778 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.308804 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:16:50.308891 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:16:50.308973 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":208,"threads_started":1}
I20260620 12:16:50.309648 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.304207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:46193 (local address 127.24.141.1:34189)
0620 12:16:50.304510 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.304523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.304710 (+ 187us) server_negotiation.cc:407] Connection header received
0620 12:16:50.304919 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.304922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.304974 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.305108 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:16:50.305599 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.306333 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.307809 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.308481 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.308626 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.308688 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.309274 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.309454 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20260620 12:16:50.310487 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb857835183a424ca7af43d61bba722e" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3adb4e587b44462a247255f73cbd38e" is_pre_election: true
I20260620 12:16:50.310530 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb857835183a424ca7af43d61bba722e" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" is_pre_election: true
I20260620 12:16:50.310940 7145 raft_consensus.cc:2454] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 0.
I20260620 12:16:50.311026 7283 raft_consensus.cc:2454] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 0.
I20260620 12:16:50.311730 7348 leader_election.cc:302] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e, f67939c8a19f4d09a9a8481534055c08; no voters:
I20260620 12:16:50.312179 7481 raft_consensus.cc:2781] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:16:50.312335 7481 raft_consensus.cc:482] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:16:50.312409 7481 raft_consensus.cc:3037] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:50.314257 7481 raft_consensus.cc:504] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.314743 7481 leader_election.cc:288] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 election: Requested vote from peers 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829), d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189)
I20260620 12:16:50.315500 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb857835183a424ca7af43d61bba722e" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3adb4e587b44462a247255f73cbd38e"
I20260620 12:16:50.315689 7145 raft_consensus.cc:3037] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:50.315665 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb857835183a424ca7af43d61bba722e" candidate_uuid: "f67939c8a19f4d09a9a8481534055c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9"
I20260620 12:16:50.315852 7283 raft_consensus.cc:3037] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:16:50.316946 7145 raft_consensus.cc:2454] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 1.
I20260620 12:16:50.317261 7283 raft_consensus.cc:2454] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f67939c8a19f4d09a9a8481534055c08 in term 1.
I20260620 12:16:50.317555 7348 leader_election.cc:302] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e, f67939c8a19f4d09a9a8481534055c08; no voters:
I20260620 12:16:50.317823 7481 raft_consensus.cc:2781] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:16:50.318128 7481 raft_consensus.cc:686] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 1 LEADER]: Becoming Leader. State: Replica: f67939c8a19f4d09a9a8481534055c08, State: Running, Role: LEADER
I20260620 12:16:50.318781 7481 consensus_queue.cc:237] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.323503 7001 catalog_manager.cc:5495] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 reported cstate change: term changed from 0 to 1, leader changed from <none> to f67939c8a19f4d09a9a8481534055c08 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "f67939c8a19f4d09a9a8481534055c08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } health_report { overall_health: UNKNOWN } } }
I20260620 12:16:50.341356 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.337193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.24.141.3:43563)
0620 12:16:50.337355 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:50.337369 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:50.337387 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:16:50.337510 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:50.337513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:50.337581 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:50.337695 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:16:50.337965 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.338573 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.339504 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:50.339748 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:50.339846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:50.340211 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:16:50.340843 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:16:50.340964 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:50.341088 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 12:16:50.341168 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 12:16:50.349661 7192 tablet.cc:2369] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:16:50.349960 7145 raft_consensus.cc:1261] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Refusing update from remote peer f67939c8a19f4d09a9a8481534055c08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:50.350060 7283 raft_consensus.cc:1261] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Refusing update from remote peer f67939c8a19f4d09a9a8481534055c08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:16:50.351006 7481 consensus_queue.cc:1035] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:16:50.351377 7485 consensus_queue.cc:1035] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:16:50.359521 7487 mvcc.cc:205] Tried to move back new op lower bound from 7298899191181594624 to 7298899191069499392. Current Snapshot: MvccSnapshot[applied={T|T < 7298899191181594624}]
I20260620 12:16:50.373600 7003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:16:50.373852 7003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:16:50.377632 7003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Sending DeleteTablet for 3 replicas of tablet eb857835183a424ca7af43d61bba722e
I20260620 12:16:50.378729 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet eb857835183a424ca7af43d61bba722e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:50 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:32854
I20260620 12:16:50.378844 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet eb857835183a424ca7af43d61bba722e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:50 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:34518
I20260620 12:16:50.379180 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet eb857835183a424ca7af43d61bba722e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:16:50 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:54620
I20260620 12:16:50.379385 7496 tablet_replica.cc:330] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: stopping tablet replica
I20260620 12:16:50.379550 7496 raft_consensus.cc:2229] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:16:50.379733 7496 raft_consensus.cc:2258] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:50.379914 7497 tablet_replica.cc:330] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: stopping tablet replica
I20260620 12:16:50.379953 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 7041
I20260620 12:16:50.381166 7497 raft_consensus.cc:2229] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:16:50.381711 7497 raft_consensus.cc:2258] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:50.383841 7497 ts_tablet_manager.cc:1905] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:16:50.387028 7497 ts_tablet_manager.cc:1918] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:16:50.387180 7497 log.cc:1196] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/eb857835183a424ca7af43d61bba722e
I20260620 12:16:50.387524 7497 ts_tablet_manager.cc:1939] T eb857835183a424ca7af43d61bba722e P 041eae0c0c9e4f89b58cdf2bc176ddb9: Deleting consensus metadata
W20260620 12:16:50.389256 6987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:34189 (error 108)
I20260620 12:16:50.389372 7498 tablet_replica.cc:330] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: stopping tablet replica
I20260620 12:16:50.389845 7498 raft_consensus.cc:2229] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:16:50.390007 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 7195
I20260620 12:16:50.390216 6988 catalog_manager.cc:4841] TS 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829): tablet eb857835183a424ca7af43d61bba722e (table test-table [id=7a316a9bcea5437693ecb97bc7e00f4c]) successfully deleted
I20260620 12:16:50.392102 7498 raft_consensus.cc:2258] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:16:50.392993 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.392532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34189 (local address 127.0.0.1:32856)
0620 12:16:50.392772 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.392859 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260620 12:16:50.393338 6987 catalog_manager.cc:4584] TS d3adb4e587b44462a247255f73cbd38e (127.24.141.1:34189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb857835183a424ca7af43d61bba722e: Network error: Client connection negotiation failed: client connection to 127.24.141.1:34189: connect: Connection refused (error 111)
I20260620 12:16:50.394305 7498 ts_tablet_manager.cc:1905] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:16:50.397485 7498 ts_tablet_manager.cc:1918] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:16:50.397645 7498 log.cc:1196] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/eb857835183a424ca7af43d61bba722e
I20260620 12:16:50.398281 7498 ts_tablet_manager.cc:1939] T eb857835183a424ca7af43d61bba722e P f67939c8a19f4d09a9a8481534055c08: Deleting consensus metadata
I20260620 12:16:50.398618 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 7332
I20260620 12:16:50.406203 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 6970
I20260620 12:16:50.414531 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45297
--webserver_interface=127.24.141.62
--webserver_port=42385
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:16:50.555230 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:50.555667 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:50.555737 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:50.560461 7500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:16:50.560602 7500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:50.560634 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:50.560657 7500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:16:50.560678 7500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:16:50.567691 7500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:50.569510 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:50.571159 7500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:50.582307 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:50.582448 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:50.582307 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:50.582677 7500 server_base.cc:1034] running on GCE node
I20260620 12:16:50.583731 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:50.585196 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:50.586380 7500 hybrid_clock.cc:648] HybridClock initialized: now 1781957810586337 us; error 62 us; skew 500 ppm
Jun 20 12:16:50 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957810, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:50.590349 7500 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:16:50.591674 7500 webserver.cc:460] Webserver started at http://127.24.141.62:42385/ using document root <none> and password file <none>
I20260620 12:16:50.592442 7500 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:50.592530 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:50.599351 7500 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:16:50.603022 7515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:50.605320 7500 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:16:50.605525 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "433a532ae71a48c8a2f56fa143752d2a"
format_stamp: "Formatted at 2026-06-20 12:16:47 on dist-test-slave-rh5v"
server_key: "b1ba99d1daa9441d94738e5f843e6c1e"
server_key_iv: "e2b89f221548afedb73f1e5040274618"
server_key_version: "encryptionkey@0"
I20260620 12:16:50.606298 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:50.628450 7500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:50.633752 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:50.634254 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:50.644716 7500 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:45297
I20260620 12:16:50.645146 7567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:45297 every 8 connection(s)
I20260620 12:16:50.646318 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:16:50.652582 7568 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:16:50.655169 7568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Bootstrap starting.
I20260620 12:16:50.655210 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 7500
I20260620 12:16:50.655684 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:34189
--local_ip_for_outbound_sockets=127.24.141.1
--tserver_master_addrs=127.24.141.62:45297
--webserver_port=39533
--webserver_interface=127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:16:50.684531 7568 log.cc:824] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:50.703022 7568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:16:50.703629 7568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a: Bootstrap complete.
I20260620 12:16:50.710166 7568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:50.710871 7568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a532ae71a48c8a2f56fa143752d2a, State: Initialized, Role: FOLLOWER
I20260620 12:16:50.711585 7568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:50.711776 7568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:16:50.711856 7568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:16:50.711957 7568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:16:50.715452 7568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:50.716019 7568 leader_election.cc:302] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 433a532ae71a48c8a2f56fa143752d2a; no voters:
I20260620 12:16:50.716528 7568 leader_election.cc:288] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 12:16:50.716790 7572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:16:50.717245 7572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [term 3 LEADER]: Becoming Leader. State: Replica: 433a532ae71a48c8a2f56fa143752d2a, State: Running, Role: LEADER
I20260620 12:16:50.717681 7572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } }
I20260620 12:16:50.718318 7568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:16:50.719491 7574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 433a532ae71a48c8a2f56fa143752d2a. Latest consensus state: current_term: 3 leader_uuid: "433a532ae71a48c8a2f56fa143752d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } } }
I20260620 12:16:50.719696 7574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:50.719491 7573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "433a532ae71a48c8a2f56fa143752d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433a532ae71a48c8a2f56fa143752d2a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 45297 } } }
I20260620 12:16:50.719825 7573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433a532ae71a48c8a2f56fa143752d2a [sys.catalog]: This master's current role is: LEADER
I20260620 12:16:50.720172 7581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:16:50.727203 7581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:16:50.728147 7581 catalog_manager.cc:1259] Loaded cluster ID: ca306f4718a74f1db7d43a85fdd34781
I20260620 12:16:50.728235 7581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 12:16:50.729611 7581 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6f107aa9ad google::LogMessage::Fail() at ??:0
@ 0x7f6f107ad88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6f107aa4de google::LogMessage::Flush() at ??:0
@ 0x7f6f107aba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6f1b0e9fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6f1b0ead2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6f1b0e7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6f1b12b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6f129fc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6f129fcc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6f129fe3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6f129ed512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6f140316db start_thread at ??:0
@ 0x7f6f0f2a861f clone at ??:0
W20260620 12:16:50.838152 7570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:50.838550 7570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:50.838619 7570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:50.843699 7570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:50.843802 7570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:50.843899 7570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:16:50.850956 7570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:34189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=39533
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:50.852638 7570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:50.854206 7570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:50.868404 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:50.868402 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:50.868402 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:50.869356 7570 server_base.cc:1034] running on GCE node
I20260620 12:16:50.869989 7570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:50.870991 7570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:50.874358 7570 hybrid_clock.cc:648] HybridClock initialized: now 1781957810874256 us; error 107 us; skew 500 ppm
Jun 20 12:16:50 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957810, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:50.878739 7570 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:16:50.880190 7570 webserver.cc:460] Webserver started at http://127.24.141.1:39533/ using document root <none> and password file <none>
I20260620 12:16:50.881016 7570 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:50.881165 7570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:50.887615 7570 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:16:50.891593 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:50.893925 7570 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 12:16:50.894181 7570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "d3adb4e587b44462a247255f73cbd38e"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "d9c93e610a47546f60d19027ac6cad4e"
server_key_iv: "4f48dba0cefb6454c498c3fa5f03c7c5"
server_key_version: "encryptionkey@0"
I20260620 12:16:50.895042 7570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:50.911221 7570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:50.915643 7570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:50.915885 7570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:50.916695 7570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:50.918299 7626 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:16:50.923931 7570 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:16:50.924041 7570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 12:16:50.924108 7570 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:16:50.926414 7570 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:16:50.926501 7570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 12:16:50.926848 7626 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Bootstrap starting.
I20260620 12:16:50.938936 7570 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:34189
I20260620 12:16:50.939157 7733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:34189 every 8 connection(s)
I20260620 12:16:50.940415 7570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:16:50.948246 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 7570
I20260620 12:16:50.948836 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:38829
--local_ip_for_outbound_sockets=127.24.141.2
--tserver_master_addrs=127.24.141.62:45297
--webserver_port=42803
--webserver_interface=127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:16:50.954689 7626 log.cc:824] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:16:50.968559 7626 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:16:50.969158 7626 tablet_bootstrap.cc:492] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Bootstrap complete.
I20260620 12:16:50.970674 7626 ts_tablet_manager.cc:1397] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Time spent bootstrapping tablet: real 0.044s user 0.033s sys 0.008s
I20260620 12:16:50.975867 7626 raft_consensus.cc:348] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.976631 7626 raft_consensus.cc:729] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3adb4e587b44462a247255f73cbd38e, State: Initialized, Role: FOLLOWER
I20260620 12:16:50.977384 7626 consensus_queue.cc:260] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:50.978897 7626 ts_tablet_manager.cc:1428] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
W20260620 12:16:50.988672 7736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:16:50.942974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:43941)
0620 12:16:50.943593 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.943658 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:50.944677 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:50.987503 (+ 42826us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":145,"threads_started":1}
W20260620 12:16:50.989686 7734 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:45297 (0 consecutive failures): Network error: Failed to ping master at 127.24.141.62:45297: Client connection negotiation failed: client connection to 127.24.141.62:45297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:16:50.990898 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.990330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:54245)
0620 12:16:50.990610 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.990780 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 12:16:50.991895 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:50.991577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:53377)
0620 12:16:50.991751 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:50.991816 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 12:16:50.992126 7734 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:16:51.092916 7737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:51.093356 7737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:51.093429 7737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:51.098249 7737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:51.098362 7737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:51.098433 7737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:16:51.105123 7737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:38829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=42803
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:51.106734 7737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:51.108188 7737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:51.121860 7747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:51.121905 7744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:51.121905 7745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:51.122407 7737 server_base.cc:1034] running on GCE node
I20260620 12:16:51.122936 7737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:51.123908 7737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:51.125185 7737 hybrid_clock.cc:648] HybridClock initialized: now 1781957811125161 us; error 45 us; skew 500 ppm
Jun 20 12:16:51 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957811, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:51.128957 7737 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:16:51.130195 7737 webserver.cc:460] Webserver started at http://127.24.141.2:42803/ using document root <none> and password file <none>
I20260620 12:16:51.130925 7737 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:51.131011 7737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:51.136373 7737 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:16:51.139690 7754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:51.141989 7737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:16:51.142207 7737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "5712a734f32c071f6ec4a8f4730ef307"
server_key_iv: "5bac9da23281cd2e427e1d899f791dbd"
server_key_version: "encryptionkey@0"
I20260620 12:16:51.143224 7737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:51.167335 7737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:51.171211 7737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:51.171442 7737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:51.172209 7737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:51.173434 7737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:51.173493 7737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:51.173558 7737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:51.173588 7737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:51.184859 7737 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:38829
I20260620 12:16:51.184926 7867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:38829 every 8 connection(s)
I20260620 12:16:51.186167 7737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:16:51.189210 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 7737
I20260620 12:16:51.189649 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:43563
--local_ip_for_outbound_sockets=127.24.141.3
--tserver_master_addrs=127.24.141.62:45297
--webserver_port=34499
--webserver_interface=127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:16:51.192025 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:51.189398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:51733)
0620 12:16:51.190177 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:51.190372 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":204,"threads_started":1}
W20260620 12:16:51.192761 7868 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:45297 (0 consecutive failures): Network error: Failed to ping master at 127.24.141.62:45297: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
I20260620 12:16:51.193948 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:51.193591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:44143)
0620 12:16:51.193791 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:51.193864 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 12:16:51.194865 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:51.194559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:34473)
0620 12:16:51.194727 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:51.194791 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 12:16:51.195094 7868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:16:51.333633 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:16:51.333981 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:16:51.334031 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:16:51.340648 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:16:51.340806 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:16:51.340889 7871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:16:51.348974 7871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:43563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=34499
--tserver_master_addrs=127.24.141.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:16:51.351042 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:16:51.352620 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:16:51.365763 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:51.365763 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:16:51.365765 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:16:51.366281 7871 server_base.cc:1034] running on GCE node
I20260620 12:16:51.366842 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:16:51.367645 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:16:51.368885 7871 hybrid_clock.cc:648] HybridClock initialized: now 1781957811368857 us; error 65 us; skew 500 ppm
Jun 20 12:16:51 dist-test-slave-rh5v krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957811, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:16:51.372704 7871 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:16:51.374322 7871 webserver.cc:460] Webserver started at http://127.24.141.3:34499/ using document root <none> and password file <none>
I20260620 12:16:51.375036 7871 fs_manager.cc:359] Metadata directory not provided
I20260620 12:16:51.375114 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:16:51.380192 7871 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:16:51.383433 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:51.385437 7871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:16:51.385646 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "f67939c8a19f4d09a9a8481534055c08"
format_stamp: "Formatted at 2026-06-20 12:16:48 on dist-test-slave-rh5v"
server_key: "c389175625f298068311018af12fad0f"
server_key_iv: "2a3f3ef2f0eefceb4b1b183810f54a45"
server_key_version: "encryptionkey@0"
I20260620 12:16:51.386346 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:16:51.405359 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:16:51.409896 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:16:51.410096 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:16:51.410761 7871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:16:51.411779 7871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:16:51.411839 7871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:51.411885 7871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:16:51.411900 7871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:16:51.424015 7871 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:43563
I20260620 12:16:51.424094 7999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:43563 every 8 connection(s)
I20260620 12:16:51.425295 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:16:51.429806 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 7871
I20260620 12:16:51.431469 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:51.428403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:36111)
0620 12:16:51.429272 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:51.429514 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":173,"threads_started":1}
W20260620 12:16:51.432344 8000 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:45297 (0 consecutive failures): Network error: Failed to ping master at 127.24.141.62:45297: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
I20260620 12:16:51.433369 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:51.433006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:60339)
0620 12:16:51.433194 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:51.433272 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 12:16:51.434382 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:51.434108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:45423)
0620 12:16:51.434265 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:51.434316 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 12:16:51.434602 8000 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:16:51.993889 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:51.993138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:48869)
0620 12:16:51.993573 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:51.993684 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":100,"threads_started":1}
I20260620 12:16:52.196691 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:52.195982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:58529)
0620 12:16:52.196451 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:52.196548 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":107,"threads_started":1}
I20260620 12:16:52.203329 8006 raft_consensus.cc:482] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:16:52.203815 8006 raft_consensus.cc:504] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:52.206073 8006 leader_election.cc:288] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563), 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829)
Jun 20 12:16:52 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957810, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.2@KRBTEST.COM
Jun 20 12:16:52 dist-test-slave-rh5v krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957810, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.3@KRBTEST.COM
I20260620 12:16:52.227592 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:52.206492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:43563 (local address 127.24.141.1:58287)
0620 12:16:52.206730 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:52.206757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:52.206930 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:52.210072 (+ 3142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:52.210079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:52.210687 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:52.211452 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:52.211481 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.213519 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.213523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:52.214887 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:52.214908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.216323 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.216327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:52.216725 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:52.217631 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:52.217652 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:52.220077 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:52.223539 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:52.223542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:52.223545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:52.223870 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:52.224758 (+ 888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:52.224761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:52.224763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:52.224901 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:52.225982 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:52.225992 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:52.226588 (+ 596us) client_negotiation.cc:769] Sending connection context
0620 12:16:52.226875 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 12:16:52.227415 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":443,"spinlock_wait_cycles":59264}
I20260620 12:16:52.227593 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:52.206837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38829 (local address 127.24.141.1:60677)
0620 12:16:52.207271 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:52.207285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:16:52.207372 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:16:52.210544 (+ 3172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:16:52.210548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:16:52.210800 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:16:52.211452 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:52.211478 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.213392 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.213398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:52.214877 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:16:52.214902 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.216172 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.216177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:16:52.216566 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:52.217519 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:16:52.217542 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:16:52.219884 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:16:52.223303 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:52.223314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:52.223334 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:52.223730 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:52.224402 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:16:52.224406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:16:52.224408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:16:52.224676 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:16:52.225982 (+ 1306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:16:52.225992 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:16:52.226588 (+ 596us) client_negotiation.cc:769] Sending connection context
0620 12:16:52.226875 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 12:16:52.227335 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":452,"spinlock_wait_cycles":4992,"thread_start_us":146,"threads_started":1}
I20260620 12:16:52.227883 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:52.207087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:60677 (local address 127.24.141.2:38829)
0620 12:16:52.207584 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:52.207607 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:52.207637 (+ 30us) server_negotiation.cc:407] Connection header received
0620 12:16:52.208641 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:52.208647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:52.209369 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:52.209854 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:52.211735 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.213146 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.215076 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.215915 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.216158 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:52.220307 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:52.220316 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:52.220337 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:52.220385 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:52.223079 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:52.223898 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:52.223908 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:52.223913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:52.224151 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:52.224840 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:52.224845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:52.224847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:52.225727 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:52.225903 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:52.227141 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0620 12:16:52.227633 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":135,"threads_started":1}
I20260620 12:16:52.229672 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:52.206820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:58287 (local address 127.24.141.3:43563)
0620 12:16:52.207439 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 12:16:52.207465 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:16:52.207534 (+ 69us) server_negotiation.cc:407] Connection header received
0620 12:16:52.208635 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:16:52.208647 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:16:52.209369 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:16:52.209854 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:16:52.211841 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.213252 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.215071 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:16:52.216132 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:16:52.216343 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:16:52.220374 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:16:52.220380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:16:52.220395 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:16:52.220440 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:16:52.223286 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:52.224024 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:52.224031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:52.224039 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:52.224212 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:16:52.225098 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:16:52.225104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:16:52.225105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:16:52.225727 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:16:52.225903 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 12:16:52.227141 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0620 12:16:52.227633 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":190,"threads_started":1}
W20260620 12:16:52.232043 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
W20260620 12:16:52.232378 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
I20260620 12:16:52.232443 7621 leader_election.cc:302] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e; no voters: 041eae0c0c9e4f89b58cdf2bc176ddb9, f67939c8a19f4d09a9a8481534055c08
I20260620 12:16:52.232858 8006 raft_consensus.cc:2726] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:16:52.436048 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:52.435350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:40977)
0620 12:16:52.435782 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:52.435881 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":103,"threads_started":1}
I20260620 12:16:52.995781 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:52.995105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:34251)
0620 12:16:52.995545 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:52.995637 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":114,"threads_started":1}
I20260620 12:16:53.198583 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:53.197941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:38187)
0620 12:16:53.198354 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:53.198433 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260620 12:16:53.437884 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:53.437225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:42505)
0620 12:16:53.437633 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:53.437721 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":105,"threads_started":1}
I20260620 12:16:53.997709 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:53.996962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:34293)
0620 12:16:53.997422 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:53.997562 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":176,"threads_started":1}
I20260620 12:16:54.200361 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:54.199726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:52993)
0620 12:16:54.200131 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:54.200222 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260620 12:16:54.285543 8016 raft_consensus.cc:482] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:16:54.285719 8016 raft_consensus.cc:504] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:54.286197 8016 leader_election.cc:288] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563), 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829)
W20260620 12:16:54.287520 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
W20260620 12:16:54.287698 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
I20260620 12:16:54.287762 7621 leader_election.cc:302] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e; no voters: 041eae0c0c9e4f89b58cdf2bc176ddb9, f67939c8a19f4d09a9a8481534055c08
I20260620 12:16:54.288045 8016 raft_consensus.cc:2726] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:16:54.439791 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:54.439117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:37465)
0620 12:16:54.439557 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:54.439641 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":106,"threads_started":1}
I20260620 12:16:54.999796 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:54.998933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:43911)
0620 12:16:54.999363 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:54.999606 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
I20260620 12:16:55.202325 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:55.201674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:41007)
0620 12:16:55.202091 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:55.202190 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260620 12:16:55.441823 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:55.440991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:55371)
0620 12:16:55.441461 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:55.441657 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":158,"threads_started":1}
I20260620 12:16:56.001909 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:56.001117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:42863)
0620 12:16:56.001566 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:56.001657 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":95,"threads_started":1}
I20260620 12:16:56.204010 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:56.203421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:42377)
0620 12:16:56.203815 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:56.203901 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260620 12:16:56.443607 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:56.443045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:52157)
0620 12:16:56.443398 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:56.443479 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":130,"threads_started":1}
I20260620 12:16:57.003604 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:57.003012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:57919)
0620 12:16:57.003359 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:57.003432 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260620 12:16:57.205470 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:57.204990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:43029)
0620 12:16:57.205305 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:57.205371 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":122,"threads_started":1}
I20260620 12:16:57.352681 8026 raft_consensus.cc:482] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:16:57.352916 8026 raft_consensus.cc:504] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:16:57.353521 8026 leader_election.cc:288] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563), 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829)
W20260620 12:16:57.354624 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
W20260620 12:16:57.354765 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
I20260620 12:16:57.354810 7621 leader_election.cc:302] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e; no voters: 041eae0c0c9e4f89b58cdf2bc176ddb9, f67939c8a19f4d09a9a8481534055c08
I20260620 12:16:57.354980 8026 raft_consensus.cc:2726] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:16:57.445303 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:57.444814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:40105)
0620 12:16:57.445111 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:57.445175 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20260620 12:16:58.005404 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:58.004762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:33767)
0620 12:16:58.005170 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:58.005259 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
I20260620 12:16:58.207036 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:58.206412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:37779)
0620 12:16:58.206829 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:58.206898 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":82,"threads_started":1}
I20260620 12:16:58.446841 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:58.446208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:45955)
0620 12:16:58.446622 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:58.446729 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":102,"threads_started":1}
I20260620 12:16:59.007146 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:59.006453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:53861)
0620 12:16:59.006901 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:59.007000 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":110,"threads_started":1}
I20260620 12:16:59.208667 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:59.208082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:51109)
0620 12:16:59.208433 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:59.208505 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260620 12:16:59.448388 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:16:59.447781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:59449)
0620 12:16:59.448143 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 12:16:59.448228 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260620 12:17:00.009614 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:00.008591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:50877)
0620 12:17:00.009022 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:00.009289 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":107,"threads_started":1}
I20260620 12:17:00.210325 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:00.209756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:44995)
0620 12:17:00.210107 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:00.210182 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260620 12:17:00.450111 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:00.449427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:51797)
0620 12:17:00.449878 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:00.449983 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":110,"threads_started":1}
I20260620 12:17:00.929508 8037 raft_consensus.cc:482] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:17:00.929711 8037 raft_consensus.cc:504] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f67939c8a19f4d09a9a8481534055c08" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 43563 } } peers { permanent_uuid: "041eae0c0c9e4f89b58cdf2bc176ddb9" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38829 } } peers { permanent_uuid: "d3adb4e587b44462a247255f73cbd38e" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34189 } }
I20260620 12:17:00.930220 8037 leader_election.cc:288] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563), 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829)
W20260620 12:17:00.931391 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f67939c8a19f4d09a9a8481534055c08 (127.24.141.3:43563): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
W20260620 12:17:00.931633 7621 leader_election.cc:341] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 041eae0c0c9e4f89b58cdf2bc176ddb9 (127.24.141.2:38829): Not found: Tablet not found: eb857835183a424ca7af43d61bba722e
I20260620 12:17:00.931699 7621 leader_election.cc:302] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3adb4e587b44462a247255f73cbd38e; no voters: 041eae0c0c9e4f89b58cdf2bc176ddb9, f67939c8a19f4d09a9a8481534055c08
I20260620 12:17:00.931906 8037 raft_consensus.cc:2726] T eb857835183a424ca7af43d61bba722e P d3adb4e587b44462a247255f73cbd38e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:17:01.011358 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:01.010798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:52973)
0620 12:17:01.011141 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:01.011223 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260620 12:17:01.212169 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:01.211544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:51847)
0620 12:17:01.211939 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:01.212033 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20260620 12:17:01.452229 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:01.451329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:55231)
0620 12:17:01.451936 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:01.452068 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":151,"threads_started":1}
I20260620 12:17:02.013303 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:02.012560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:54687)
0620 12:17:02.013010 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:02.013163 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":134,"threads_started":1}
I20260620 12:17:02.214059 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:02.213317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:54059)
0620 12:17:02.213792 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:02.213908 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":105,"threads_started":1}
I20260620 12:17:02.454223 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:02.453598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:55151)
0620 12:17:02.454026 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:02.454095 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":135,"threads_started":1}
I20260620 12:17:03.015113 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:03.014575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:39395)
0620 12:17:03.014942 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:03.015004 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":105,"threads_started":1}
I20260620 12:17:03.215828 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:03.215213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:41001)
0620 12:17:03.215591 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:03.215703 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260620 12:17:03.456053 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:03.455353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:40841)
0620 12:17:03.455832 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:03.455912 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":233,"threads_started":1}
I20260620 12:17:04.016788 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:04.016157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:45465)
0620 12:17:04.016524 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:04.016650 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260620 12:17:04.217558 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:04.216918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:45831)
0620 12:17:04.217366 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:04.217448 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
I20260620 12:17:04.457896 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:04.457122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:36901)
0620 12:17:04.457643 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:04.457753 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":121,"threads_started":1}
I20260620 12:17:05.018764 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:05.018022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:46429)
0620 12:17:05.018522 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:05.018611 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":129,"threads_started":1}
I20260620 12:17:05.219442 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:05.218784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:52105)
0620 12:17:05.219225 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:05.219325 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":111,"threads_started":1}
I20260620 12:17:05.459760 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:05.458931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.3:38443)
0620 12:17:05.459406 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:05.459527 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":102,"threads_started":1}
I20260620 12:17:06.020709 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:06.020130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.1:38289)
0620 12:17:06.020493 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:06.020576 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260620 12:17:06.221487 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:06.220786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:45297 (local address 127.24.141.2:38669)
0620 12:17:06.221252 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:06.221344 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":124,"threads_started":1}
I20260620 12:17:06.436930 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 7570
I20260620 12:17:06.443804 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 7737
I20260620 12:17:06.450289 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 7871
2026-06-20T12:17:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19858 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:17:06 dist-test-slave-rh5v krb5kdc[8057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:17:06 dist-test-slave-rh5v krb5kdc[8057](info): set up 2 sockets
Jun 20 12:17:06 dist-test-slave-rh5v krb5kdc[8057](info): commencing operation
krb5kdc: starting...
W20260620 12:17:07.438253 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:17:07 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:17:07.589886 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40207
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:40207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:17:07.743294 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:07.743789 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:07.743841 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:07.748870 8070 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:17:07.748966 8070 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:17:07.748983 8070 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:17:07.748996 8070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:17:07.749017 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:07.749035 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:07.749050 8070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:17:07.749110 8070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:17:07.756096 8070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:40207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:07.757881 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:07.759750 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:07.773051 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:07.773031 8075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:07.773031 8078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:07.773783 8070 server_base.cc:1034] running on GCE node
I20260620 12:17:07.774713 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:07.776686 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:07.777915 8070 hybrid_clock.cc:648] HybridClock initialized: now 1781957827777895 us; error 53 us; skew 500 ppm
Jun 20 12:17:07 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957827, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:07.785871 8070 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:17:07.787175 8070 webserver.cc:460] Webserver started at http://127.24.141.62:38563/ using document root <none> and password file <none>
I20260620 12:17:07.787845 8070 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:07.787919 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:07.788125 8070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:17:07.790648 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "9756cfc3547c42c5bf8041d658f88d0d"
format_stamp: "Formatted at 2026-06-20 12:17:07 on dist-test-slave-rh5v"
server_key: "a66b058ecfa6081073e575c85e825f6b"
server_key_iv: "5ee77ffd3c422af9c177817caa620f52"
server_key_version: "encryptionkey@0"
I20260620 12:17:07.791290 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "9756cfc3547c42c5bf8041d658f88d0d"
format_stamp: "Formatted at 2026-06-20 12:17:07 on dist-test-slave-rh5v"
server_key: "a66b058ecfa6081073e575c85e825f6b"
server_key_iv: "5ee77ffd3c422af9c177817caa620f52"
server_key_version: "encryptionkey@0"
I20260620 12:17:07.796386 8070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:17:07.800350 8085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:07.802831 8070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 12:17:07.803051 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "9756cfc3547c42c5bf8041d658f88d0d"
format_stamp: "Formatted at 2026-06-20 12:17:07 on dist-test-slave-rh5v"
server_key: "a66b058ecfa6081073e575c85e825f6b"
server_key_iv: "5ee77ffd3c422af9c177817caa620f52"
server_key_version: "encryptionkey@0"
I20260620 12:17:07.803231 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:07.820447 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:07.825315 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:07.825555 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:07.834766 8070 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:40207
I20260620 12:17:07.835155 8137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:40207 every 8 connection(s)
I20260620 12:17:07.836052 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:17:07.840360 8138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:17:07.840967 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8070
I20260620 12:17:07.841395 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:17:07.841900 25140 external_mini_cluster.cc:1428] Setting key 8c412fa4e58c223a59cf5fe274a87541
I20260620 12:17:07.850409 8138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Bootstrap starting.
I20260620 12:17:07.854931 8138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Neither blocks nor log segments found. Creating new log.
I20260620 12:17:07.856376 8138 log.cc:824] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:17:07 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:17:07.860131 8138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: No bootstrap required, opened a new log
I20260620 12:17:07.866019 8138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:07.866532 8138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:17:07.866650 8138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9756cfc3547c42c5bf8041d658f88d0d, State: Initialized, Role: FOLLOWER
I20260620 12:17:07.867195 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:07.844156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.24.141.62:40207)
0620 12:17:07.844778 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:07.844797 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:07.844842 (+ 45us) server_negotiation.cc:407] Connection header received
0620 12:17:07.845836 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:07.845863 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:07.846616 (+ 753us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:07.847112 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:07.848415 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:07.849831 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:07.851670 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:07.853683 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:07.853929 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:07.858274 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:07.858283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:07.858305 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:07.858353 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:07.860779 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:07.861711 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:07.861724 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:07.861731 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:07.861956 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:07.862414 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:07.862419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:07.862420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:07.863047 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:07.863264 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:07.864154 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 12:17:07.865168 (+ 1014us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":181,"threads_started":1}
I20260620 12:17:07.867291 8138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:07.867516 8138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:17:07.867728 8138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:17:07.867851 8138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:17:07.869661 8138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:07.870239 8138 leader_election.cc:302] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9756cfc3547c42c5bf8041d658f88d0d; no voters:
I20260620 12:17:07.870733 8138 leader_election.cc:288] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:17:07.870889 8143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:17:07.871176 8143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 LEADER]: Becoming Leader. State: Replica: 9756cfc3547c42c5bf8041d658f88d0d, State: Running, Role: LEADER
I20260620 12:17:07.871596 8143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:07.872696 8138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:17:07.874184 8144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9756cfc3547c42c5bf8041d658f88d0d. Latest consensus state: current_term: 1 leader_uuid: "9756cfc3547c42c5bf8041d658f88d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } } }
I20260620 12:17:07.874490 8145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9756cfc3547c42c5bf8041d658f88d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } } }
I20260620 12:17:07.874580 8145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:17:07.874486 8144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:17:07.875463 8153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:17:07.879998 8153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:17:07.888289 8153 catalog_manager.cc:1347] Generated new cluster ID: 14a7a408502f479c855ce46a31350ffa
I20260620 12:17:07.888548 8153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:17:07.912918 8153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:17:07.914592 8153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:17:07.927536 8153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Generated new TSK 0
I20260620 12:17:07.929790 8153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:17:07.972537 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40207
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:17:08.119864 8164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:08.120263 8164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:08.120316 8164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:08.125332 8164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:08.125454 8164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:08.125531 8164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:17:08.133400 8164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:08.135316 8164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:08.136974 8164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:08.150998 8170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:08.151060 8169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:08.151060 8172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:08.151580 8164 server_base.cc:1034] running on GCE node
I20260620 12:17:08.152284 8164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:08.153240 8164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:08.154507 8164 hybrid_clock.cc:648] HybridClock initialized: now 1781957828154486 us; error 46 us; skew 500 ppm
Jun 20 12:17:08 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:08.158329 8164 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:17:08.159569 8164 webserver.cc:460] Webserver started at http://127.24.141.1:42111/ using document root <none> and password file <none>
I20260620 12:17:08.160254 8164 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:08.160336 8164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:08.160569 8164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:17:08.163161 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "c71aa47f95254554a6cfa10a8117ccd3"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "558f98dc3ed1dbe0f4d8a6479579ec59"
server_key_iv: "dd95135283aa69e03f60a0926496ff4d"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.163872 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "c71aa47f95254554a6cfa10a8117ccd3"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "558f98dc3ed1dbe0f4d8a6479579ec59"
server_key_iv: "dd95135283aa69e03f60a0926496ff4d"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.170032 8164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 12:17:08.174202 8179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.176359 8164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:17:08.176577 8164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "c71aa47f95254554a6cfa10a8117ccd3"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "558f98dc3ed1dbe0f4d8a6479579ec59"
server_key_iv: "dd95135283aa69e03f60a0926496ff4d"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.176949 8164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:08.197973 8164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:08.202070 8164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:08.202316 8164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:08.203073 8164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:08.204137 8164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:17:08.204198 8164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.204244 8164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:17:08.204258 8164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.216025 8164 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:38519
I20260620 12:17:08.216097 8292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:38519 every 8 connection(s)
I20260620 12:17:08.217208 8164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:17:08.223170 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8164
I20260620 12:17:08.223313 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:17:08.223591 25140 external_mini_cluster.cc:1428] Setting key 7fa5b2f614fbf1cadef28c6dbf53c673
Jun 20 12:17:08 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:17:08.237675 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.220011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:53755 (local address 127.24.141.62:40207)
0620 12:17:08.220204 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.220208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.221419 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 12:17:08.223169 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.223174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.223255 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.223352 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:08.225267 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.225858 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.227766 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.228247 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.228403 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.232331 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:08.232336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:08.232339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:08.232380 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:08.234102 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.235310 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.235315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.235316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.235437 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.236172 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.236176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.236178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.236452 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:08.236562 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.237271 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.237483 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:17:08.238631 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.220502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:53755)
0620 12:17:08.221197 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.221262 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.222857 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.223727 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.223741 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.224313 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:08.225053 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.225104 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.226069 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.226074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.227531 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.227546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.228519 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.228524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.228763 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.229583 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:08.229608 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:08.231942 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:08.234494 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:08.234505 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:08.234530 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:08.235028 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:08.235763 (+ 735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:08.235767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:08.235769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:08.236022 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:08.236587 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:08.236597 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:08.236940 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.237183 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.237594 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":170,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:17:08.240564 8293 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40207
I20260620 12:17:08.241109 8293 heartbeater.cc:464] Registering TS with master...
I20260620 12:17:08.241921 8293 heartbeater.cc:511] Master 127.24.141.62:40207 requested a full tablet report, sending...
I20260620 12:17:08.244802 8099 ts_manager.cc:194] Registered new tserver with Master: c71aa47f95254554a6cfa10a8117ccd3 (127.24.141.1:38519)
I20260620 12:17:08.246306 8099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:53755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:17:08.257161 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40207
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:17:08.403172 8298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:08.403582 8298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:08.403636 8298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:08.408241 8298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:08.408347 8298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:08.408418 8298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:17:08.415004 8298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:08.416738 8298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:08.418315 8298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:08.431140 8303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:08.431170 8304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:08.431504 8298 server_base.cc:1034] running on GCE node
W20260620 12:17:08.431702 8306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:08.432322 8298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:08.433225 8298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:08.434491 8298 hybrid_clock.cc:648] HybridClock initialized: now 1781957828434466 us; error 46 us; skew 500 ppm
Jun 20 12:17:08 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:08.437876 8298 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:17:08.439421 8298 webserver.cc:460] Webserver started at http://127.24.141.2:46071/ using document root <none> and password file <none>
I20260620 12:17:08.440289 8298 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:08.440377 8298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:08.440644 8298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:17:08.443789 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "2869af5c64024e0db4e0eaeffe13df35"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "84c705ef54287eba044959fec5e781d4"
server_key_iv: "7633cd90b5f43b6f8fccb4a577766588"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.444697 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "2869af5c64024e0db4e0eaeffe13df35"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "84c705ef54287eba044959fec5e781d4"
server_key_iv: "7633cd90b5f43b6f8fccb4a577766588"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.449967 8298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 12:17:08.453822 8313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.455842 8298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:17:08.456039 8298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "2869af5c64024e0db4e0eaeffe13df35"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "84c705ef54287eba044959fec5e781d4"
server_key_iv: "7633cd90b5f43b6f8fccb4a577766588"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.456215 8298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:08.499524 8298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:08.503309 8298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:08.503512 8298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:08.504256 8298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:08.505435 8298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:17:08.505503 8298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.505568 8298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:17:08.505601 8298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.516980 8298 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:33401
I20260620 12:17:08.517084 8426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:33401 every 8 connection(s)
I20260620 12:17:08.518211 8298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:17:08.528633 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8298
I20260620 12:17:08.528777 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:17:08.529105 25140 external_mini_cluster.cc:1428] Setting key aeed2fc57e0254902e6373d4efcdabfe
Jun 20 12:17:08 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:17:08.537806 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.520870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:46733 (local address 127.24.141.62:40207)
0620 12:17:08.521130 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.521136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.522299 (+ 1163us) server_negotiation.cc:407] Connection header received
0620 12:17:08.523990 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.523994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.524069 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.524186 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:08.525882 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.526577 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.528025 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.528370 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.528557 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.532418 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:08.532424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:08.532428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:08.532469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:08.534438 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.535487 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.535491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.535493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.535648 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.536140 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.536143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.536145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.536435 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:08.536556 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.537413 (+ 857us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.537612 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 12:17:08.538751 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.521414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:46733)
0620 12:17:08.522059 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.522124 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.523636 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.524408 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.524417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.524933 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:08.525642 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.525660 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.526742 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.526746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.527776 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.527788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.528722 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.528728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.528958 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.529774 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:08.529795 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:08.532131 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:08.534677 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:08.534688 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:08.534713 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:08.535290 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:08.535799 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:08.535803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:08.535804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:08.536009 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:08.536577 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:08.536591 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:08.536964 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.537250 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.537681 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":146,"threads_started":1}
I20260620 12:17:08.540762 8427 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40207
I20260620 12:17:08.541253 8427 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:17:08.542156 8427 heartbeater.cc:511] Master 127.24.141.62:40207 requested a full tablet report, sending...
I20260620 12:17:08.544039 8099 ts_manager.cc:194] Registered new tserver with Master: 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:17:08.545033 8099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:46733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:17:08.562485 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40207
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:17:08.713640 8432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:08.714061 8432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:08.714111 8432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:08.718719 8432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:08.718820 8432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:08.718891 8432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:17:08.725317 8432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:08.726852 8432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:08.728160 8432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:08.741124 8438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:08.741151 8440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:08.741124 8437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:08.741602 8432 server_base.cc:1034] running on GCE node
I20260620 12:17:08.742218 8432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:08.743121 8432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:08.744318 8432 hybrid_clock.cc:648] HybridClock initialized: now 1781957828744291 us; error 54 us; skew 500 ppm
Jun 20 12:17:08 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:08.748180 8432 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:17:08.749651 8432 webserver.cc:460] Webserver started at http://127.24.141.3:42895/ using document root <none> and password file <none>
I20260620 12:17:08.750437 8432 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:08.750524 8432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:08.750792 8432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:17:08.753665 8432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "1927ab05b8d44dfaba4c061e8c4ab049"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "67b7f5eb6b052cf1c56ebf81c640fa48"
server_key_iv: "f0261b984e0f0f45ffeb9decf2e1b574"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.754396 8432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "1927ab05b8d44dfaba4c061e8c4ab049"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "67b7f5eb6b052cf1c56ebf81c640fa48"
server_key_iv: "f0261b984e0f0f45ffeb9decf2e1b574"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.760043 8432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:17:08.764591 8447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.766880 8432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:08.767145 8432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "1927ab05b8d44dfaba4c061e8c4ab049"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "67b7f5eb6b052cf1c56ebf81c640fa48"
server_key_iv: "f0261b984e0f0f45ffeb9decf2e1b574"
server_key_version: "encryptionkey@0"
I20260620 12:17:08.767339 8432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:08.789008 8432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:08.793751 8432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:08.793988 8432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:08.794771 8432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:08.796015 8432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:17:08.796085 8432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.796154 8432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:17:08.796181 8432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:08.808223 8432 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:36239
I20260620 12:17:08.808244 8560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:36239 every 8 connection(s)
I20260620 12:17:08.809562 8432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:17:08.813053 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8432
I20260620 12:17:08.813251 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:17:08.813663 25140 external_mini_cluster.cc:1428] Setting key 4d9ddfc1412f06dbef4495abec6ad062
Jun 20 12:17:08 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:17:08.828573 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.812217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:33297 (local address 127.24.141.62:40207)
0620 12:17:08.812427 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.812433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.814087 (+ 1654us) server_negotiation.cc:407] Connection header received
0620 12:17:08.815941 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.815945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.816024 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.816114 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:08.817894 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.818473 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.819803 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.820093 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.820205 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.823863 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:08.823869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:08.823871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:08.823914 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:08.825523 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.826225 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.826228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.826230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.826340 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.826903 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.826906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.826908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.827173 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:08.827286 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.828177 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.828381 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 12:17:08.829841 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.812802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:33297)
0620 12:17:08.813866 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.813942 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.815662 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.816558 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.816567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.817089 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:08.817685 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.817702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.818625 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.818628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.819645 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.819658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.820372 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.820379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.820745 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.821595 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:08.821617 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:08.823534 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:08.825677 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:08.825683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:08.825703 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:08.826026 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:08.826443 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:08.826447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:08.826449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:08.826684 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:08.827307 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:08.827316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:08.827757 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.828028 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.828493 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":241,"threads_started":1}
I20260620 12:17:08.831446 8561 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40207
I20260620 12:17:08.831833 8561 heartbeater.cc:464] Registering TS with master...
I20260620 12:17:08.832729 8561 heartbeater.cc:511] Master 127.24.141.62:40207 requested a full tablet report, sending...
I20260620 12:17:08.834483 8099 ts_manager.cc:194] Registered new tserver with Master: 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
I20260620 12:17:08.835433 8099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:33297
I20260620 12:17:08.842500 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:17:08.856559 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.845949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.24.141.62:40207)
0620 12:17:08.846145 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.846151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.846401 (+ 250us) server_negotiation.cc:407] Connection header received
0620 12:17:08.846545 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.846549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.846615 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.846742 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:08.847541 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.848400 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.849573 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.849839 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.849966 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.852329 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:08.852334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:08.852336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:08.852380 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:08.854067 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.854845 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.854849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.854850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.854953 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:08.855355 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:08.855357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:08.855358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:08.855680 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:08.855814 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.856186 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.856338 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:17:08.860782 8099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:17:08.865005 8099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:17:08.887881 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.881219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33401 (local address 127.0.0.1:54018)
0620 12:17:08.881654 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.881670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.881783 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.882577 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.882580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.882596 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:08.882728 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.882735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.884432 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.884436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.885808 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.885820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.887110 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.887114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.887278 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.887327 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.887470 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.887662 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":110,"thread_start_us":114,"threads_started":1}
I20260620 12:17:08.887914 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.881671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.24.141.2:33401)
0620 12:17:08.882069 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.882082 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.882109 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:17:08.882206 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.882211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.882416 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.882582 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:08.882833 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.884220 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.885954 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.886920 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.887069 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.887293 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.887531 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.887693 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":182,"threads_started":1}
I20260620 12:17:08.888026 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.880265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.24.141.1:38519)
0620 12:17:08.880748 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.880756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.881393 (+ 637us) server_negotiation.cc:407] Connection header received
0620 12:17:08.881678 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.881684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.881962 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.882111 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:08.883045 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.883993 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.886228 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.887100 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.887216 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.887342 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.887749 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.887876 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":88,"threads_started":1}
I20260620 12:17:08.888233 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.879850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36239 (local address 127.0.0.1:36448)
0620 12:17:08.881187 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.881269 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.881475 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.882418 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.882428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.882468 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:08.882722 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.882732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.884432 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.884436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.886192 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.886211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.887495 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.887499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.887662 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.887689 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.887769 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.887845 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":144,"thread_start_us":135,"threads_started":1}
I20260620 12:17:08.888751 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.880475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.24.141.3:36239)
0620 12:17:08.880922 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.880932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.881635 (+ 703us) server_negotiation.cc:407] Connection header received
0620 12:17:08.881712 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.881717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.881953 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.882110 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:08.882952 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.884227 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.886368 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.887111 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.887242 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.887428 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.887970 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.888613 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":185,"threads_started":1}
I20260620 12:17:08.889356 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.879867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:38519 (local address 127.0.0.1:53554)
0620 12:17:08.881187 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.881269 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.881475 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.882474 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.882477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.882492 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:08.882744 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.882751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.884158 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.884163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.885958 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.885973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.887412 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.887415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.887547 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.887611 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.889139 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.889246 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":103,"spinlock_wait_cycles":27136,"thread_start_us":147,"threads_started":1}
I20260620 12:17:08.892345 8495 tablet_service.cc:1467] Processing CreateTablet for tablet a171866831e6431da986589897f3a6c5 (DEFAULT_TABLE table=test-table [id=6662a11c368a4e5d871d0b90a7a7ce1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:17:08.892480 8361 tablet_service.cc:1467] Processing CreateTablet for tablet a171866831e6431da986589897f3a6c5 (DEFAULT_TABLE table=test-table [id=6662a11c368a4e5d871d0b90a7a7ce1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:17:08.892480 8227 tablet_service.cc:1467] Processing CreateTablet for tablet a171866831e6431da986589897f3a6c5 (DEFAULT_TABLE table=test-table [id=6662a11c368a4e5d871d0b90a7a7ce1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:17:08.893898 8495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a171866831e6431da986589897f3a6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:17:08.893978 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a171866831e6431da986589897f3a6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:17:08.893978 8361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a171866831e6431da986589897f3a6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:17:08.901908 8578 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Bootstrap starting.
I20260620 12:17:08.904222 8580 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap starting.
I20260620 12:17:08.904424 8579 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Bootstrap starting.
I20260620 12:17:08.905826 8578 tablet_bootstrap.cc:654] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Neither blocks nor log segments found. Creating new log.
I20260620 12:17:08.907187 8578 log.cc:824] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:08.907466 8580 tablet_bootstrap.cc:654] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Neither blocks nor log segments found. Creating new log.
I20260620 12:17:08.907959 8579 tablet_bootstrap.cc:654] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Neither blocks nor log segments found. Creating new log.
I20260620 12:17:08.908766 8580 log.cc:824] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:08.908766 8579 log.cc:824] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:08.909844 8578 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: No bootstrap required, opened a new log
I20260620 12:17:08.910120 8578 ts_tablet_manager.cc:1397] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:17:08.911412 8580 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: No bootstrap required, opened a new log
I20260620 12:17:08.911674 8580 ts_tablet_manager.cc:1397] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:17:08.911690 8579 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: No bootstrap required, opened a new log
I20260620 12:17:08.912361 8579 ts_tablet_manager.cc:1397] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 12:17:08.914494 8578 raft_consensus.cc:348] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.915081 8578 raft_consensus.cc:374] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:17:08.915189 8578 raft_consensus.cc:729] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1927ab05b8d44dfaba4c061e8c4ab049, State: Initialized, Role: FOLLOWER
I20260620 12:17:08.916191 8578 consensus_queue.cc:260] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.916239 8580 raft_consensus.cc:348] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.916591 8580 raft_consensus.cc:374] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:17:08.916643 8580 raft_consensus.cc:729] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71aa47f95254554a6cfa10a8117ccd3, State: Initialized, Role: FOLLOWER
I20260620 12:17:08.916756 8579 raft_consensus.cc:348] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.917258 8579 raft_consensus.cc:374] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:17:08.917353 8579 raft_consensus.cc:729] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2869af5c64024e0db4e0eaeffe13df35, State: Initialized, Role: FOLLOWER
I20260620 12:17:08.917425 8561 heartbeater.cc:503] Master 127.24.141.62:40207 was elected leader, sending a full tablet report...
I20260620 12:17:08.917352 8580 consensus_queue.cc:260] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.917798 8578 ts_tablet_manager.cc:1428] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 12:17:08.918000 8579 consensus_queue.cc:260] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.918648 8580 ts_tablet_manager.cc:1428] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 12:17:08.918983 8293 heartbeater.cc:503] Master 127.24.141.62:40207 was elected leader, sending a full tablet report...
I20260620 12:17:08.918987 8427 heartbeater.cc:503] Master 127.24.141.62:40207 was elected leader, sending a full tablet report...
I20260620 12:17:08.919153 8579 ts_tablet_manager.cc:1428] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 12:17:08.969137 8294 tablet.cc:2369] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:17:08.973694 8585 raft_consensus.cc:482] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:17:08.973989 8585 raft_consensus.cc:504] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.976109 8585 leader_election.cc:288] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239), 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401)
I20260620 12:17:08.981308 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.976608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:60831 (local address 127.24.141.3:36239)
0620 12:17:08.976911 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.976916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.977087 (+ 171us) server_negotiation.cc:407] Connection header received
0620 12:17:08.977286 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.977290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.977341 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.977423 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:08.977790 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.978425 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.979848 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.980500 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.980637 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.980687 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.981014 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.981149 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260620 12:17:08.981335 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.976444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36239 (local address 127.24.141.1:60831)
0620 12:17:08.976904 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.976922 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.977155 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.977453 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.977457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.977479 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:08.977670 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.977679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.978552 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.978556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.979634 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.979649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.980683 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.980687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.980805 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.980842 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.980970 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.981104 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":160,"threads_started":1}
I20260620 12:17:08.981452 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.976788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:49033 (local address 127.24.141.2:33401)
0620 12:17:08.976935 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:08.976940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:08.977290 (+ 350us) server_negotiation.cc:407] Connection header received
0620 12:17:08.977513 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:08.977516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:08.977584 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:08.977675 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:08.978052 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.978700 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.980115 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.980669 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.980819 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.980880 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:08.981212 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 12:17:08.981318 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:17:08.981582 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:08.976645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33401 (local address 127.24.141.1:49033)
0620 12:17:08.977148 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:08.977163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:08.977330 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:08.977722 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:08.977725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:08.977747 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:08.977916 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.977924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.978898 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.978902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.979846 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:08.979859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:08.980858 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:08.980869 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:08.981015 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:08.981088 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 12:17:08.981182 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 12:17:08.981300 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":143,"threads_started":1}
I20260620 12:17:08.982457 8381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171866831e6431da986589897f3a6c5" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2869af5c64024e0db4e0eaeffe13df35" is_pre_election: true
I20260620 12:17:08.982458 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171866831e6431da986589897f3a6c5" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" is_pre_election: true
I20260620 12:17:08.982930 8381 raft_consensus.cc:2454] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 0.
I20260620 12:17:08.982930 8515 raft_consensus.cc:2454] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 0.
I20260620 12:17:08.983657 8181 leader_election.cc:302] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927ab05b8d44dfaba4c061e8c4ab049, c71aa47f95254554a6cfa10a8117ccd3; no voters:
I20260620 12:17:08.984004 8585 raft_consensus.cc:2781] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:17:08.984107 8585 raft_consensus.cc:482] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:17:08.984140 8585 raft_consensus.cc:3037] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:17:08.985514 8585 raft_consensus.cc:504] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.986011 8585 leader_election.cc:288] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 election: Requested vote from peers 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239), 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401)
I20260620 12:17:08.986588 8381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171866831e6431da986589897f3a6c5" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2869af5c64024e0db4e0eaeffe13df35"
I20260620 12:17:08.986590 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171866831e6431da986589897f3a6c5" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1927ab05b8d44dfaba4c061e8c4ab049"
I20260620 12:17:08.986744 8381 raft_consensus.cc:3037] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:17:08.986800 8515 raft_consensus.cc:3037] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:17:08.988118 8515 raft_consensus.cc:2454] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 1.
I20260620 12:17:08.988198 8381 raft_consensus.cc:2454] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 1.
I20260620 12:17:08.988601 8181 leader_election.cc:302] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927ab05b8d44dfaba4c061e8c4ab049, c71aa47f95254554a6cfa10a8117ccd3; no voters:
I20260620 12:17:08.988917 8585 raft_consensus.cc:2781] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:17:08.989235 8585 raft_consensus.cc:686] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 1 LEADER]: Becoming Leader. State: Replica: c71aa47f95254554a6cfa10a8117ccd3, State: Running, Role: LEADER
I20260620 12:17:08.989725 8585 consensus_queue.cc:237] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:08.994323 8095 catalog_manager.cc:5495] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c71aa47f95254554a6cfa10a8117ccd3 (127.24.141.1). New cstate: current_term: 1 leader_uuid: "c71aa47f95254554a6cfa10a8117ccd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } health_report { overall_health: UNKNOWN } } }
I20260620 12:17:09.018157 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:09.013857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.24.141.1:38519)
0620 12:17:09.014170 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:09.014175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:09.014201 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:17:09.014299 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:09.014302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:09.014389 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:09.014509 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:17:09.014840 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:09.015398 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:09.016437 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:09.016688 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:09.016784 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:09.017211 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:17:09.017742 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:17:09.017858 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:09.017929 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 12:17:09.017999 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260620 12:17:09.019701 8428 tablet.cc:2369] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:17:09.025884 8381 raft_consensus.cc:1261] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Refusing update from remote peer c71aa47f95254554a6cfa10a8117ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:17:09.025884 8515 raft_consensus.cc:1261] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Refusing update from remote peer c71aa47f95254554a6cfa10a8117ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:17:09.026772 8585 consensus_queue.cc:1035] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:17:09.027081 8589 consensus_queue.cc:1035] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:17:09.037851 8594 mvcc.cc:205] Tried to move back new op lower bound from 7298899267679936512 to 7298899267545346048. Current Snapshot: MvccSnapshot[applied={T|T < 7298899267679936512}]
I20260620 12:17:09.038846 8593 mvcc.cc:205] Tried to move back new op lower bound from 7298899267679936512 to 7298899267545346048. Current Snapshot: MvccSnapshot[applied={T|T < 7298899267679936512}]
I20260620 12:17:09.048894 8095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:17:09.049182 8095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:17:09.052120 8095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Sending DeleteTablet for 3 replicas of tablet a171866831e6431da986589897f3a6c5
I20260620 12:17:09.053442 8495 tablet_service.cc:1514] Processing DeleteTablet for tablet a171866831e6431da986589897f3a6c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:17:09 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:36448
I20260620 12:17:09.053442 8361 tablet_service.cc:1514] Processing DeleteTablet for tablet a171866831e6431da986589897f3a6c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:17:09 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:54018
I20260620 12:17:09.054140 8602 tablet_replica.cc:330] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: stopping tablet replica
I20260620 12:17:09.054425 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8164
I20260620 12:17:09.054399 8227 tablet_service.cc:1514] Processing DeleteTablet for tablet a171866831e6431da986589897f3a6c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:17:09 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:53554
I20260620 12:17:09.056061 8602 raft_consensus.cc:2229] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:17:09.056528 8602 raft_consensus.cc:2258] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:17:09.056687 8603 tablet_replica.cc:330] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: stopping tablet replica
I20260620 12:17:09.057189 8603 raft_consensus.cc:2229] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:17:09.057726 8603 raft_consensus.cc:2258] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:17:09.058570 8602 ts_tablet_manager.cc:1905] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:17:09.059871 8603 ts_tablet_manager.cc:1905] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:17:09.062031 8602 ts_tablet_manager.cc:1918] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:17:09.062186 8602 log.cc:1196] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/a171866831e6431da986589897f3a6c5
I20260620 12:17:09.063113 8602 ts_tablet_manager.cc:1939] T a171866831e6431da986589897f3a6c5 P 2869af5c64024e0db4e0eaeffe13df35: Deleting consensus metadata
I20260620 12:17:09.063127 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8298
I20260620 12:17:09.063444 8603 ts_tablet_manager.cc:1918] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:17:09.063621 8603 log.cc:1196] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/a171866831e6431da986589897f3a6c5
I20260620 12:17:09.064002 8603 ts_tablet_manager.cc:1939] T a171866831e6431da986589897f3a6c5 P 1927ab05b8d44dfaba4c061e8c4ab049: Deleting consensus metadata
W20260620 12:17:09.065449 8089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:38519 (error 108)
I20260620 12:17:09.065801 8087 catalog_manager.cc:4841] TS 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239): tablet a171866831e6431da986589897f3a6c5 (table test-table [id=6662a11c368a4e5d871d0b90a7a7ce1e]) successfully deleted
I20260620 12:17:09.066875 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:09.066458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:38519 (local address 127.0.0.1:53560)
0620 12:17:09.066634 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:09.066762 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 12:17:09.067119 8089 catalog_manager.cc:4584] TS c71aa47f95254554a6cfa10a8117ccd3 (127.24.141.1:38519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a171866831e6431da986589897f3a6c5: Network error: Client connection negotiation failed: client connection to 127.24.141.1:38519: connect: Connection refused (error 111)
I20260620 12:17:09.071404 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8432
I20260620 12:17:09.072808 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:09.071743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33401 (local address 127.0.0.1:54028)
0620 12:17:09.072545 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:09.072625 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.2:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":682}
I20260620 12:17:09.079192 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8070
I20260620 12:17:29.087934 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40207
--webserver_interface=127.24.141.62
--webserver_port=38563
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:40207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:17:29.217932 8605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:29.218269 8605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:29.218315 8605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:29.222702 8605 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:17:29.222797 8605 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:17:29.222815 8605 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:17:29.222829 8605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:17:29.222849 8605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:29.222867 8605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:29.222882 8605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:17:29.222898 8605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:17:29.230026 8605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:40207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:29.231637 8605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:29.233101 8605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:29.252136 8614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:29.252054 8612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:29.252199 8605 server_base.cc:1034] running on GCE node
W20260620 12:17:29.252053 8611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:29.253355 8605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:29.254819 8605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:29.256069 8605 hybrid_clock.cc:648] HybridClock initialized: now 1781957849256023 us; error 64 us; skew 500 ppm
Jun 20 12:17:29 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:29.259733 8605 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:17:29.260993 8605 webserver.cc:460] Webserver started at http://127.24.141.62:38563/ using document root <none> and password file <none>
I20260620 12:17:29.261791 8605 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:29.261899 8605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:29.268381 8605 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:17:29.272151 8621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:29.274390 8605 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:29.274609 8605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "9756cfc3547c42c5bf8041d658f88d0d"
format_stamp: "Formatted at 2026-06-20 12:17:07 on dist-test-slave-rh5v"
server_key: "a66b058ecfa6081073e575c85e825f6b"
server_key_iv: "5ee77ffd3c422af9c177817caa620f52"
server_key_version: "encryptionkey@0"
I20260620 12:17:29.275372 8605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:29.291189 8605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:29.295186 8605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:29.295462 8605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:29.304033 8605 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:40207
I20260620 12:17:29.304392 8673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:40207 every 8 connection(s)
I20260620 12:17:29.305491 8605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:17:29.307296 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8605
I20260620 12:17:29.307817 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:38519
--local_ip_for_outbound_sockets=127.24.141.1
--tserver_master_addrs=127.24.141.62:40207
--webserver_port=42111
--webserver_interface=127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:17:29.312242 8674 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:17:29.314630 8674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Bootstrap starting.
I20260620 12:17:29.335894 8674 log.cc:824] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:29.347337 8674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:17:29.347796 8674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Bootstrap complete.
I20260620 12:17:29.352545 8674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:29.353077 8674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9756cfc3547c42c5bf8041d658f88d0d, State: Initialized, Role: FOLLOWER
I20260620 12:17:29.353729 8674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:29.353873 8674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:17:29.353955 8674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:17:29.354058 8674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:17:29.356479 8674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:29.356877 8674 leader_election.cc:302] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9756cfc3547c42c5bf8041d658f88d0d; no voters:
I20260620 12:17:29.357276 8674 leader_election.cc:288] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:17:29.357393 8678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:17:29.357621 8678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 LEADER]: Becoming Leader. State: Replica: 9756cfc3547c42c5bf8041d658f88d0d, State: Running, Role: LEADER
I20260620 12:17:29.358050 8678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:29.358819 8674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:17:29.360059 8680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9756cfc3547c42c5bf8041d658f88d0d. Latest consensus state: current_term: 2 leader_uuid: "9756cfc3547c42c5bf8041d658f88d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } } }
I20260620 12:17:29.360318 8680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:17:29.360679 8679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9756cfc3547c42c5bf8041d658f88d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } } }
I20260620 12:17:29.360806 8679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:17:29.361342 8688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:17:29.369764 8688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:17:29.370748 8688 catalog_manager.cc:1259] Loaded cluster ID: 14a7a408502f479c855ce46a31350ffa
I20260620 12:17:29.370844 8688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:17:29.373824 8688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:17:29.374850 8688 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Loaded TSK: 0
I20260620 12:17:29.389223 8688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Generated new TSK 1
I20260620 12:17:29.389680 8688 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:17:29.444181 8676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:29.444511 8676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:29.444576 8676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:29.448978 8676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:29.449105 8676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:29.449193 8676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:17:29.455042 8676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:38519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=42111
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:29.456336 8676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:29.457561 8676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:29.472597 8702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:29.472708 8676 server_base.cc:1034] running on GCE node
W20260620 12:17:29.472609 8704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:29.472929 8701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:29.473910 8676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:29.475185 8676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:29.476450 8676 hybrid_clock.cc:648] HybridClock initialized: now 1781957849476411 us; error 62 us; skew 500 ppm
Jun 20 12:17:29 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:29.481449 8676 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:17:29.483080 8676 webserver.cc:460] Webserver started at http://127.24.141.1:42111/ using document root <none> and password file <none>
I20260620 12:17:29.484321 8676 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:29.484506 8676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:29.491412 8676 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:17:29.495149 8711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:29.497362 8676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:29.497597 8676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "c71aa47f95254554a6cfa10a8117ccd3"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "558f98dc3ed1dbe0f4d8a6479579ec59"
server_key_iv: "dd95135283aa69e03f60a0926496ff4d"
server_key_version: "encryptionkey@0"
I20260620 12:17:29.498713 8676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:29.528200 8676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:29.533304 8676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:29.533599 8676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:29.534511 8676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:29.536324 8718 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:17:29.541355 8676 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:17:29.541529 8676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260620 12:17:29.541616 8676 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:17:29.544663 8676 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:17:29.544770 8676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260620 12:17:29.545218 8718 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap starting.
I20260620 12:17:29.557750 8676 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:38519
I20260620 12:17:29.558200 8825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:38519 every 8 connection(s)
I20260620 12:17:29.559259 8676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:17:29.567457 8718 log.cc:824] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:29.568439 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8676
I20260620 12:17:29.568953 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:33401
--local_ip_for_outbound_sockets=127.24.141.2
--tserver_master_addrs=127.24.141.62:40207
--webserver_port=46071
--webserver_interface=127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:17:29 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:17:29.577983 8718 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:17:29.578547 8718 tablet_bootstrap.cc:492] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap complete.
I20260620 12:17:29.580073 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:29.562631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:39583 (local address 127.24.141.62:40207)
0620 12:17:29.563144 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:29.563173 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:29.563731 (+ 558us) server_negotiation.cc:407] Connection header received
0620 12:17:29.565283 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:29.565296 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:29.565723 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:29.566096 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:29.567570 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.568819 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.570286 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.570939 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.571129 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:29.574309 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:29.574315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:29.574331 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:29.574368 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:29.576240 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:29.576815 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:29.576821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:29.576825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:29.576949 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:29.577327 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:29.577330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:29.577331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:29.577781 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:29.577915 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:29.578587 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 12:17:29.578955 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":166,"threads_started":1}
I20260620 12:17:29.580084 8718 ts_tablet_manager.cc:1397] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.007s
I20260620 12:17:29.580101 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:29.562724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:39583)
0620 12:17:29.563483 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:29.563556 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:29.564667 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:29.566219 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:29.566230 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:29.566785 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:29.567357 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:29.567376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.569009 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.569013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:29.570137 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:29.570151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.571165 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.571171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:29.571436 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:29.572292 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:29.572314 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:29.574065 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:29.576376 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:29.576384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:29.576401 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:29.576694 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:29.577051 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:29.577078 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:29.577080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:29.577233 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:29.577901 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:29.577908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:29.578237 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 12:17:29.578426 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:17:29.578881 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":242,"threads_started":1}
I20260620 12:17:29.583151 8826 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40207
I20260620 12:17:29.583467 8826 heartbeater.cc:464] Registering TS with master...
I20260620 12:17:29.584125 8826 heartbeater.cc:511] Master 127.24.141.62:40207 requested a full tablet report, sending...
I20260620 12:17:29.584412 8718 raft_consensus.cc:348] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:29.585314 8718 raft_consensus.cc:729] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71aa47f95254554a6cfa10a8117ccd3, State: Initialized, Role: FOLLOWER
I20260620 12:17:29.586017 8718 consensus_queue.cc:260] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } } peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } }
I20260620 12:17:29.587630 8718 ts_tablet_manager.cc:1428] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:17:29.590648 8638 ts_manager.cc:194] Registered new tserver with Master: c71aa47f95254554a6cfa10a8117ccd3 (127.24.141.1:38519)
I20260620 12:17:29.595679 8638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:39583
I20260620 12:17:29.599391 8826 heartbeater.cc:503] Master 127.24.141.62:40207 was elected leader, sending a full tablet report...
Jun 20 12:17:29 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.1@KRBTEST.COM
I20260620 12:17:29.609970 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:29.595205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:38519 (local address 127.0.0.1:50472)
0620 12:17:29.595986 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:29.596053 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:29.596250 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:29.597142 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:29.597154 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:29.597742 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:29.597927 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:29.597937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.599525 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.599529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:29.600422 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:29.600437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.601390 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.601393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:29.601517 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:29.602399 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:29.602413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:29.604421 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:29.607163 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:29.607184 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:29.607192 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:29.607702 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:29.608158 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:29.608162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:29.608164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:29.608292 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:29.609049 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:29.609091 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:29.609284 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 12:17:29.609470 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 12:17:29.609756 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":188,"threads_started":1}
I20260620 12:17:29.610035 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:29.595310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.24.141.1:38519)
0620 12:17:29.596534 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:29.596546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:29.596572 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:17:29.596653 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:29.596658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:29.596915 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:29.597109 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:29.598122 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.599323 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.600580 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.601211 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.601343 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:29.604721 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:29.604736 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:29.604753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:29.604795 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:29.606946 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:29.607898 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:29.607907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:29.607909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:29.608037 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:29.608495 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:29.608499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:29.608500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:29.608893 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:29.609119 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:29.609627 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 12:17:29.609848 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":124,"threads_started":1}
I20260620 12:17:29.611474 8760 tablet_service.cc:1514] Processing DeleteTablet for tablet a171866831e6431da986589897f3a6c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:17:09 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:50472
I20260620 12:17:29.612582 8835 tablet_replica.cc:330] stopping tablet replica
I20260620 12:17:29.613173 8835 raft_consensus.cc:2229] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:17:29.613559 8835 raft_consensus.cc:2258] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:17:29.614981 8835 ts_tablet_manager.cc:1905] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:17:29.619776 8835 ts_tablet_manager.cc:1918] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:17:29.619908 8835 log.cc:1196] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal/wals/a171866831e6431da986589897f3a6c5
I20260620 12:17:29.620157 8835 ts_tablet_manager.cc:1939] T a171866831e6431da986589897f3a6c5 P c71aa47f95254554a6cfa10a8117ccd3: Deleting consensus metadata
I20260620 12:17:29.622727 8625 catalog_manager.cc:4841] TS c71aa47f95254554a6cfa10a8117ccd3 (127.24.141.1:38519): tablet a171866831e6431da986589897f3a6c5 (table test-table [id=6662a11c368a4e5d871d0b90a7a7ce1e]) successfully deleted
W20260620 12:17:29.710498 8831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:29.710801 8831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:29.710846 8831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:29.715253 8831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:29.715345 8831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:29.715408 8831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:17:29.721325 8831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:33401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=46071
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:29.722515 8831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:29.723625 8831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:29.735873 8841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:29.735936 8843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:29.735936 8840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:29.736373 8831 server_base.cc:1034] running on GCE node
I20260620 12:17:29.736902 8831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:29.737833 8831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:29.739084 8831 hybrid_clock.cc:648] HybridClock initialized: now 1781957849739058 us; error 40 us; skew 500 ppm
Jun 20 12:17:29 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:29.742534 8831 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:17:29.743760 8831 webserver.cc:460] Webserver started at http://127.24.141.2:46071/ using document root <none> and password file <none>
I20260620 12:17:29.744441 8831 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:29.744529 8831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:29.749269 8831 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:17:29.752103 8850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:29.754060 8831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:29.754256 8831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "2869af5c64024e0db4e0eaeffe13df35"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "84c705ef54287eba044959fec5e781d4"
server_key_iv: "7633cd90b5f43b6f8fccb4a577766588"
server_key_version: "encryptionkey@0"
I20260620 12:17:29.754923 8831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:29.783550 8831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:29.787665 8831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:29.787909 8831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:29.788563 8831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:29.789567 8831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:17:29.789623 8831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:29.789667 8831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:17:29.789682 8831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:29.800457 8831 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:33401
I20260620 12:17:29.800537 8963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:33401 every 8 connection(s)
I20260620 12:17:29.801744 8831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:17:29.809270 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8831
I20260620 12:17:29.809901 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:36239
--local_ip_for_outbound_sockets=127.24.141.3
--tserver_master_addrs=127.24.141.62:40207
--webserver_port=42895
--webserver_interface=127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:17:29 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:17:29.821274 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:29.804434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:55709 (local address 127.24.141.62:40207)
0620 12:17:29.804700 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:29.804707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:29.805959 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 12:17:29.807753 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:29.807758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:29.807849 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:29.807938 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:29.809995 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.810788 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.812716 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.813004 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.813193 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:29.816574 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:29.816579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:29.816581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:29.816618 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:29.818484 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:29.819275 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:29.819278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:29.819280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:29.819374 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:29.819773 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:29.819777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:29.819779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:29.820029 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:29.820146 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:29.820810 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 12:17:29.821030 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 12:17:29.822451 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:29.805000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:55709)
0620 12:17:29.805763 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:29.805824 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:29.807442 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:29.808209 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:29.808219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:29.808732 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:29.809672 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:29.809703 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.811054 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.811058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:29.812521 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:29.812537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:29.813267 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:29.813273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:29.813515 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:29.814310 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:29.814331 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:29.816284 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:29.818687 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:29.818697 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:29.818725 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:29.819151 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:29.819484 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:29.819487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:29.819490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:29.819671 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:29.820152 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:29.820161 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:29.820530 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 12:17:29.820750 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 12:17:29.821270 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":185,"threads_started":1}
I20260620 12:17:29.824297 8964 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40207
I20260620 12:17:29.824645 8964 heartbeater.cc:464] Registering TS with master...
I20260620 12:17:29.825415 8964 heartbeater.cc:511] Master 127.24.141.62:40207 requested a full tablet report, sending...
I20260620 12:17:29.827023 8638 ts_manager.cc:194] Registered new tserver with Master: 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401)
I20260620 12:17:29.827955 8638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:55709
W20260620 12:17:29.943179 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:29.943511 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:29.943560 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:29.948354 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:29.948467 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:29.948535 8967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:17:29.954622 8967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:36239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=42895
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:29.956084 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:29.957263 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:29.970505 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:29.970536 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:29.970510 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:29.970629 8967 server_base.cc:1034] running on GCE node
I20260620 12:17:29.971369 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:29.972137 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:29.973399 8967 hybrid_clock.cc:648] HybridClock initialized: now 1781957849973362 us; error 44 us; skew 500 ppm
Jun 20 12:17:29 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:29.976657 8967 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:17:29.977805 8967 webserver.cc:460] Webserver started at http://127.24.141.3:42895/ using document root <none> and password file <none>
I20260620 12:17:29.978441 8967 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:29.978540 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:29.983400 8967 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:17:29.986246 8982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:29.988227 8967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:29.988430 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "1927ab05b8d44dfaba4c061e8c4ab049"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "67b7f5eb6b052cf1c56ebf81c640fa48"
server_key_iv: "f0261b984e0f0f45ffeb9decf2e1b574"
server_key_version: "encryptionkey@0"
I20260620 12:17:29.989151 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:30.002928 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:30.012356 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:30.012579 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:30.013294 8967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:30.014298 8967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:17:30.014353 8967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:30.014415 8967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:17:30.014444 8967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:30.030750 8967 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:36239
I20260620 12:17:30.030879 9095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:36239 every 8 connection(s)
I20260620 12:17:30.032047 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:17:30.040125 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 8967
Jun 20 12:17:30 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957849, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:17:30.051437 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.041223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.24.141.62:40207)
0620 12:17:30.041619 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.041623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.041880 (+ 257us) server_negotiation.cc:407] Connection header received
0620 12:17:30.042070 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.042073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.042146 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.042268 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:30.043288 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.044097 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.045199 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.045463 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.045565 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.047371 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:30.047377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:30.047380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:30.047427 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:30.049491 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:30.049996 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:30.050000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:30.050002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:30.050121 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:30.050399 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:30.050402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:30.050404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:30.050638 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:30.050891 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.051026 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.051189 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":91,"threads_started":1}
I20260620 12:17:30.052834 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.035067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:42889 (local address 127.24.141.62:40207)
0620 12:17:30.035313 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.035317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.036896 (+ 1579us) server_negotiation.cc:407] Connection header received
0620 12:17:30.038581 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.038585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.038660 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.038742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:30.040356 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.040987 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.043179 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.043569 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.043727 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.047889 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:30.047893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:30.047897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:30.047951 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:30.049666 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:30.050674 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:30.050679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:30.050681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:30.050818 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:30.051246 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:30.051250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:30.051252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:30.051577 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:30.051690 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.052525 (+ 835us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.052675 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 12:17:30.054292 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.035719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:42889)
0620 12:17:30.036642 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:30.036723 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:30.038280 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:30.038994 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:30.039003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:30.039453 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:30.040039 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.040056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.041496 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.041500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.042988 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.043006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.043817 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.043823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.044051 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.045389 (+ 1338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:30.045413 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:30.047611 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:30.049810 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:30.049819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:30.049846 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:30.050474 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:30.050938 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:30.050941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:30.050943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:30.051129 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:30.051712 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:30.051723 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:30.052226 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 12:17:30.052474 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 12:17:30.052995 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":210,"threads_started":1}
I20260620 12:17:30.055706 9096 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40207
I20260620 12:17:30.056049 9096 heartbeater.cc:464] Registering TS with master...
I20260620 12:17:30.056748 9096 heartbeater.cc:511] Master 127.24.141.62:40207 requested a full tablet report, sending...
I20260620 12:17:30.058372 8638 ts_manager.cc:194] Registered new tserver with Master: 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
I20260620 12:17:30.059177 8638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:42889
I20260620 12:17:30.067226 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:17:30.072043 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.068074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.24.141.62:40207)
0620 12:17:30.068323 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.068328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.068537 (+ 209us) server_negotiation.cc:407] Connection header received
0620 12:17:30.068769 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.068772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.068902 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.069010 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:17:30.069337 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.069878 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.070786 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.071006 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.071092 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.071431 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:17:30.071594 (+ 163us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 12:17:30.071883 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40484: token expired
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 12:17:30.072135 8828 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40484: token expired
W20260620 12:17:30.072801 8566 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.141.62:40207)
I20260620 12:17:30.082497 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.073483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.24.141.62:40207)
0620 12:17:30.073666 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.073670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.073688 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:17:30.073791 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.073793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.073841 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.073929 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:30.074766 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.075324 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.076336 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.076573 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.076685 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.078337 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:30.078341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:30.078343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:30.078381 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:30.080156 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:30.080915 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:30.080919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:30.080920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:30.081053 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:30.081446 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:30.081449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:30.081451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:30.081699 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:30.081827 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.082157 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.082295 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:17:30.086061 8638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:17:30.090066 8638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:17:30.101871 8760 tablet_service.cc:1467] Processing CreateTablet for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f (DEFAULT_TABLE table=test-table [id=806bdd3bf1e04965ad45180b06f1ee15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:17:30.102813 8760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:17:30.106019 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.099878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33401 (local address 127.0.0.1:45928)
0620 12:17:30.100119 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:30.100137 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:30.100240 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:30.101233 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:30.101236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:30.101257 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:30.101387 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.101395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.102776 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.102779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.104011 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.104022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.105443 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.105447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.105578 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.105648 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 12:17:30.105729 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:17:30.105843 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 12:17:30.106124 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.100099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.24.141.2:33401)
0620 12:17:30.100594 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.100608 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.100638 (+ 30us) server_negotiation.cc:407] Connection header received
0620 12:17:30.100740 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.100747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.101027 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.101232 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:30.101496 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.102663 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.104185 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.105287 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.105500 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.105713 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.105781 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.105921 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":205,"threads_started":1}
I20260620 12:17:30.106389 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.100190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36239 (local address 127.0.0.1:46070)
0620 12:17:30.100740 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:30.100758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:30.100883 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:30.101786 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:30.101789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:30.101818 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:30.101915 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.101921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.103639 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.103642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.104688 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.104700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.106013 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.106016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.106104 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.106127 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:17:30.106174 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 12:17:30.106253 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":163,"threads_started":1}
I20260620 12:17:30.107158 9108 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap starting.
I20260620 12:17:30.108042 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.100448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.24.141.3:36239)
0620 12:17:30.101194 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.101209 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.101238 (+ 29us) server_negotiation.cc:407] Connection header received
0620 12:17:30.101336 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.101343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.101627 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.102000 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:30.102223 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.103497 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.104855 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.105870 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.107466 (+ 1596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.107658 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.107724 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.107855 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":192,"threads_started":1}
I20260620 12:17:30.108556 9108 tablet_bootstrap.cc:654] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Neither blocks nor log segments found. Creating new log.
I20260620 12:17:30.110054 9108 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: No bootstrap required, opened a new log
I20260620 12:17:30.110189 9108 ts_tablet_manager.cc:1397] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:17:30.110965 9108 raft_consensus.cc:348] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.111119 9108 raft_consensus.cc:374] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:17:30.111158 9108 raft_consensus.cc:729] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71aa47f95254554a6cfa10a8117ccd3, State: Initialized, Role: FOLLOWER
I20260620 12:17:30.111320 9108 consensus_queue.cc:260] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.110709 8898 tablet_service.cc:1467] Processing CreateTablet for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f (DEFAULT_TABLE table=test-table [id=806bdd3bf1e04965ad45180b06f1ee15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:17:30.111720 9108 ts_tablet_manager.cc:1428] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:17:30.111496 9030 tablet_service.cc:1467] Processing CreateTablet for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f (DEFAULT_TABLE table=test-table [id=806bdd3bf1e04965ad45180b06f1ee15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:17:30.112325 8898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:17:30.112836 9030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:17:30.121518 9111 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Bootstrap starting.
I20260620 12:17:30.122426 9112 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Bootstrap starting.
I20260620 12:17:30.124727 9111 tablet_bootstrap.cc:654] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Neither blocks nor log segments found. Creating new log.
I20260620 12:17:30.125494 9112 tablet_bootstrap.cc:654] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Neither blocks nor log segments found. Creating new log.
I20260620 12:17:30.126092 9111 log.cc:824] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:30.126397 9112 log.cc:824] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:30.129096 9112 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: No bootstrap required, opened a new log
I20260620 12:17:30.129089 9111 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: No bootstrap required, opened a new log
I20260620 12:17:30.129369 9112 ts_tablet_manager.cc:1397] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:17:30.129374 9111 ts_tablet_manager.cc:1397] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:17:30.134583 9112 raft_consensus.cc:348] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.134583 9111 raft_consensus.cc:348] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.135093 9111 raft_consensus.cc:374] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:17:30.135093 9112 raft_consensus.cc:374] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:17:30.135155 9111 raft_consensus.cc:729] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1927ab05b8d44dfaba4c061e8c4ab049, State: Initialized, Role: FOLLOWER
I20260620 12:17:30.135155 9112 raft_consensus.cc:729] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2869af5c64024e0db4e0eaeffe13df35, State: Initialized, Role: FOLLOWER
I20260620 12:17:30.136022 9112 consensus_queue.cc:260] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.135991 9111 consensus_queue.cc:260] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.137166 9110 raft_consensus.cc:482] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:17:30.137475 8964 heartbeater.cc:503] Master 127.24.141.62:40207 was elected leader, sending a full tablet report...
I20260620 12:17:30.137419 9110 raft_consensus.cc:504] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.137681 9112 ts_tablet_manager.cc:1428] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 12:17:30.137681 9111 ts_tablet_manager.cc:1428] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 12:17:30.138356 9096 heartbeater.cc:503] Master 127.24.141.62:40207 was elected leader, sending a full tablet report...
I20260620 12:17:30.138875 9110 leader_election.cc:288] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401), 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
I20260620 12:17:30.144944 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.139096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33401 (local address 127.24.141.1:59063)
0620 12:17:30.139631 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:30.139647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:30.139774 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:30.140402 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:30.140406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:30.140443 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:30.140598 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.140606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.141905 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.141910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.143275 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.143292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.144392 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.144396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.144493 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.144547 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 12:17:30.144617 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 12:17:30.144730 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":118,"threads_started":1}
I20260620 12:17:30.145023 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.139575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:33263 (local address 127.24.141.3:36239)
0620 12:17:30.139728 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.139733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.140121 (+ 388us) server_negotiation.cc:407] Connection header received
0620 12:17:30.140345 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.140349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.140405 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.140478 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:30.141422 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.142237 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.143737 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.144223 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.144325 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.144399 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.144787 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.144903 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:17:30.144992 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.139851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:59063 (local address 127.24.141.2:33401)
0620 12:17:30.139979 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.139984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.139999 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:17:30.140062 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.140065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.140133 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.140229 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:17:30.140761 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.141648 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.143436 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.144065 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.144181 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.144253 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.144714 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.144838 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:17:30.145002 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.139184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36239 (local address 127.24.141.1:33263)
0620 12:17:30.140035 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:30.140048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:30.140192 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:30.140870 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:30.140874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:30.141114 (+ 240us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:17:30.141274 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.141283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.142410 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.142414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.143565 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:30.143581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.144503 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.144506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:30.144605 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.144635 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:17:30.144711 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 12:17:30.144818 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260620 12:17:30.145952 8918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eda1fd6ec3146f4b6a7ba6d5eb0957f" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2869af5c64024e0db4e0eaeffe13df35" is_pre_election: true
I20260620 12:17:30.146220 9050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eda1fd6ec3146f4b6a7ba6d5eb0957f" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" is_pre_election: true
I20260620 12:17:30.146379 8918 raft_consensus.cc:2454] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 0.
I20260620 12:17:30.146495 9050 raft_consensus.cc:2454] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 0.
I20260620 12:17:30.147058 8715 leader_election.cc:302] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2869af5c64024e0db4e0eaeffe13df35, c71aa47f95254554a6cfa10a8117ccd3; no voters:
I20260620 12:17:30.147469 9110 raft_consensus.cc:2781] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:17:30.147620 9110 raft_consensus.cc:482] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:17:30.147679 9110 raft_consensus.cc:3037] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:17:30.148840 9110 raft_consensus.cc:504] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.149653 9110 leader_election.cc:288] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 election: Requested vote from peers 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401), 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
I20260620 12:17:30.150187 9050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eda1fd6ec3146f4b6a7ba6d5eb0957f" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1927ab05b8d44dfaba4c061e8c4ab049"
I20260620 12:17:30.150400 9050 raft_consensus.cc:3037] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:17:30.151978 9050 raft_consensus.cc:2454] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 1.
I20260620 12:17:30.152606 8918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eda1fd6ec3146f4b6a7ba6d5eb0957f" candidate_uuid: "c71aa47f95254554a6cfa10a8117ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2869af5c64024e0db4e0eaeffe13df35"
I20260620 12:17:30.152740 8713 leader_election.cc:302] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927ab05b8d44dfaba4c061e8c4ab049, c71aa47f95254554a6cfa10a8117ccd3; no voters:
I20260620 12:17:30.152802 8918 raft_consensus.cc:3037] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:17:30.153053 9110 raft_consensus.cc:2781] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:17:30.153386 9110 raft_consensus.cc:686] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 LEADER]: Becoming Leader. State: Replica: c71aa47f95254554a6cfa10a8117ccd3, State: Running, Role: LEADER
I20260620 12:17:30.153764 9110 consensus_queue.cc:237] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:30.154633 8918 raft_consensus.cc:2454] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71aa47f95254554a6cfa10a8117ccd3 in term 1.
I20260620 12:17:30.157778 8638 catalog_manager.cc:5495] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c71aa47f95254554a6cfa10a8117ccd3 (127.24.141.1). New cstate: current_term: 1 leader_uuid: "c71aa47f95254554a6cfa10a8117ccd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } health_report { overall_health: HEALTHY } } }
I20260620 12:17:30.172323 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.168267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.24.141.1:38519)
0620 12:17:30.168734 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:30.168743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:30.168757 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:17:30.168848 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:30.168851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:30.168911 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:30.169032 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:17:30.169344 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.169870 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.170734 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:30.170964 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:30.171056 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:30.171352 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:17:30.171867 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:17:30.171971 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:30.172068 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 12:17:30.172141 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":188,"threads_started":1}
I20260620 12:17:30.179066 8918 raft_consensus.cc:1261] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Refusing update from remote peer c71aa47f95254554a6cfa10a8117ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:17:30.179219 9050 raft_consensus.cc:1261] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Refusing update from remote peer c71aa47f95254554a6cfa10a8117ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:17:30.179982 9110 consensus_queue.cc:1035] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:17:30.180413 9119 consensus_queue.cc:1035] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:17:30.190934 9124 mvcc.cc:205] Tried to move back new op lower bound from 7298899354324000768 to 7298899354233696256. Current Snapshot: MvccSnapshot[applied={T|T < 7298899354324000768}]
I20260620 12:17:30.192037 9123 mvcc.cc:205] Tried to move back new op lower bound from 7298899354324000768 to 7298899354233696256. Current Snapshot: MvccSnapshot[applied={T|T < 7298899354324000768}]
I20260620 12:17:30.202569 8637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:17:30.202832 8637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:17:30.206768 8637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Sending DeleteTablet for 3 replicas of tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:17:30.207979 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8676
I20260620 12:17:30.207991 9030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:17:30 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:46070
I20260620 12:17:30.208060 8898 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:17:30 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:45928
I20260620 12:17:30.209221 9133 tablet_replica.cc:330] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: stopping tablet replica
I20260620 12:17:30.209671 9133 raft_consensus.cc:2229] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:17:30.209915 9132 tablet_replica.cc:330] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: stopping tablet replica
I20260620 12:17:30.210122 9133 raft_consensus.cc:2258] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:17:30.210219 9132 raft_consensus.cc:2229] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:17:30.210529 9132 raft_consensus.cc:2258] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:17:30.212149 9132 ts_tablet_manager.cc:1905] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:17:30.212369 9133 ts_tablet_manager.cc:1905] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:17:30.215034 9132 ts_tablet_manager.cc:1918] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:17:30.215180 9132 log.cc:1196] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:17:30.215543 9132 ts_tablet_manager.cc:1939] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 1927ab05b8d44dfaba4c061e8c4ab049: Deleting consensus metadata
I20260620 12:17:30.216275 9133 ts_tablet_manager.cc:1918] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:17:30.216429 9133 log.cc:1196] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:17:30.216809 9133 ts_tablet_manager.cc:1939] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P 2869af5c64024e0db4e0eaeffe13df35: Deleting consensus metadata
I20260620 12:17:30.217082 8623 catalog_manager.cc:4841] TS 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239): tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f (table test-table [id=806bdd3bf1e04965ad45180b06f1ee15]) successfully deleted
W20260620 12:17:30.218959 8625 connection.cc:535] client connection to 127.24.141.1:38519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 12:17:30.219163 8625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:17:30.219352 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8831
I20260620 12:17:30.219703 8625 catalog_manager.cc:4841] TS 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401): tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f (table test-table [id=806bdd3bf1e04965ad45180b06f1ee15]) successfully deleted
I20260620 12:17:30.220901 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:30.220513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:38519 (local address 127.0.0.1:50500)
0620 12:17:30.220700 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:30.220790 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 12:17:30.221208 8625 catalog_manager.cc:4584] TS c71aa47f95254554a6cfa10a8117ccd3 (127.24.141.1:38519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6eda1fd6ec3146f4b6a7ba6d5eb0957f: Network error: Client connection negotiation failed: client connection to 127.24.141.1:38519: connect: Connection refused (error 111)
I20260620 12:17:30.227567 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8967
I20260620 12:17:30.234709 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 8605
I20260620 12:17:50.244143 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40207
--webserver_interface=127.24.141.62
--webserver_port=38563
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:40207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:17:50.393373 9135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:50.393814 9135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:50.393865 9135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:50.398501 9135 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:17:50.398607 9135 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:17:50.398623 9135 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:17:50.398638 9135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:17:50.398658 9135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:50.398676 9135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:50.398692 9135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:17:50.398707 9135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:17:50.405285 9135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:40207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:50.406960 9135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:50.408357 9135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:50.421099 9141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:50.421121 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:50.421202 9135 server_base.cc:1034] running on GCE node
W20260620 12:17:50.421121 9143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:50.422329 9135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:50.423954 9135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:50.427474 9135 hybrid_clock.cc:648] HybridClock initialized: now 1781957870427354 us; error 87 us; skew 500 ppm
Jun 20 12:17:50 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957870, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:50.436652 9135 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:17:50.438079 9135 webserver.cc:460] Webserver started at http://127.24.141.62:38563/ using document root <none> and password file <none>
I20260620 12:17:50.438789 9135 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:50.438871 9135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:50.444422 9135 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:17:50.448057 9150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:50.450054 9135 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:50.450285 9135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "9756cfc3547c42c5bf8041d658f88d0d"
format_stamp: "Formatted at 2026-06-20 12:17:07 on dist-test-slave-rh5v"
server_key: "a66b058ecfa6081073e575c85e825f6b"
server_key_iv: "5ee77ffd3c422af9c177817caa620f52"
server_key_version: "encryptionkey@0"
I20260620 12:17:50.451000 9135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:50.471596 9135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:50.475484 9135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:50.475729 9135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:50.484220 9135 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:40207
I20260620 12:17:50.484671 9202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:40207 every 8 connection(s)
I20260620 12:17:50.485622 9135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:17:50.492156 9203 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:17:50.494784 9203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Bootstrap starting.
I20260620 12:17:50.495442 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 9135
I20260620 12:17:50.496140 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:38519
--local_ip_for_outbound_sockets=127.24.141.1
--tserver_master_addrs=127.24.141.62:40207
--webserver_port=42111
--webserver_interface=127.24.141.1
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:17:50.519907 9203 log.cc:824] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:17:50.536885 9203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:17:50.537495 9203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Bootstrap complete.
I20260620 12:17:50.542963 9203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:50.543653 9203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9756cfc3547c42c5bf8041d658f88d0d, State: Initialized, Role: FOLLOWER
I20260620 12:17:50.544427 9203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:50.544606 9203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:17:50.544687 9203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:17:50.544806 9203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:17:50.547653 9203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:50.548132 9203 leader_election.cc:302] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9756cfc3547c42c5bf8041d658f88d0d; no voters:
I20260620 12:17:50.548545 9203 leader_election.cc:288] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 12:17:50.548699 9207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:17:50.548925 9207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [term 3 LEADER]: Becoming Leader. State: Replica: 9756cfc3547c42c5bf8041d658f88d0d, State: Running, Role: LEADER
I20260620 12:17:50.549417 9207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } }
I20260620 12:17:50.550295 9203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:17:50.551399 9209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9756cfc3547c42c5bf8041d658f88d0d. Latest consensus state: current_term: 3 leader_uuid: "9756cfc3547c42c5bf8041d658f88d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } } }
I20260620 12:17:50.551594 9209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:17:50.551491 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9756cfc3547c42c5bf8041d658f88d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9756cfc3547c42c5bf8041d658f88d0d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40207 } } }
I20260620 12:17:50.551740 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d [sys.catalog]: This master's current role is: LEADER
I20260620 12:17:50.552263 9216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:17:50.559470 9216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:17:50.560458 9216 catalog_manager.cc:1259] Loaded cluster ID: 14a7a408502f479c855ce46a31350ffa
I20260620 12:17:50.560537 9216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:17:50.563066 9216 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:17:50.564002 9216 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Loaded TSK: 0
I20260620 12:17:50.564110 9216 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9756cfc3547c42c5bf8041d658f88d0d: Loaded TSK: 1
F20260620 12:17:50.565042 9216 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1b23fdd9ad google::LogMessage::Fail() at ??:0
@ 0x7f1b23fe088a google::LogMessage::SendToLog() at ??:0
@ 0x7f1b23fdd4de google::LogMessage::Flush() at ??:0
@ 0x7f1b23fdea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1b26ca118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1b26c99f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1b2e9427bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1b2e91c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1b2e91d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f1b2e95edf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1b2e97d82b std::function<>::operator()() at ??:0
@ 0x7f1b2e91cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1b2e91df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1b2e91a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1b2e95e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1b2622f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1b2622fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1b262313f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1b26220512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1b278646db start_thread at ??:0
@ 0x7f1b22adb61f clone at ??:0
W20260620 12:17:50.687703 9205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:50.688055 9205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:50.688102 9205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:50.692907 9205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:50.693025 9205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:50.693152 9205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:17:50.699584 9205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:38519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=42111
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:50.700920 9205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:50.702176 9205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:50.718166 9250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:50.718256 9205 server_base.cc:1034] running on GCE node
W20260620 12:17:50.718155 9251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:50.718601 9253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:50.719133 9205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:50.719970 9205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:50.721192 9205 hybrid_clock.cc:648] HybridClock initialized: now 1781957870721186 us; error 94 us; skew 500 ppm
Jun 20 12:17:50 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957870, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:50.725977 9205 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:17:50.727802 9205 webserver.cc:460] Webserver started at http://127.24.141.1:42111/ using document root <none> and password file <none>
I20260620 12:17:50.728657 9205 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:50.728775 9205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:50.734251 9205 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:50.737916 9260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:50.740249 9205 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:17:50.740502 9205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "c71aa47f95254554a6cfa10a8117ccd3"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "558f98dc3ed1dbe0f4d8a6479579ec59"
server_key_iv: "dd95135283aa69e03f60a0926496ff4d"
server_key_version: "encryptionkey@0"
I20260620 12:17:50.741457 9205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:50.782068 9205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:50.786679 9205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:50.786940 9205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:50.787801 9205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:50.789383 9267 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:17:50.795370 9205 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:17:50.795499 9205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 12:17:50.795571 9205 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:17:50.798280 9205 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:17:50.798382 9205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260620 12:17:50.798739 9267 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap starting.
I20260620 12:17:50.811249 9205 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:38519
I20260620 12:17:50.812719 9374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:38519 every 8 connection(s)
I20260620 12:17:50.813221 9205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:17:50.818260 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 9205
I20260620 12:17:50.818941 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:33401
--local_ip_for_outbound_sockets=127.24.141.2
--tserver_master_addrs=127.24.141.62:40207
--webserver_port=46071
--webserver_interface=127.24.141.2
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:17:50.831351 9267 log.cc:824] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Log is configured to *not* fsync() on all Append() calls
W20260620 12:17:50.838941 9377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:17:50.817131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:43111)
0620 12:17:50.819337 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:50.819450 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:50.821222 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:50.837479 (+ 16257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1756,"thread_start_us":293,"threads_started":1}
W20260620 12:17:50.839752 9375 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:40207 (0 consecutive failures): Network error: Failed to ping master at 127.24.141.62:40207: Client connection negotiation failed: client connection to 127.24.141.62:40207: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:17:50.840873 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:50.840405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:45913)
0620 12:17:50.840625 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:50.840743 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 12:17:50.842063 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:50.841726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:58805)
0620 12:17:50.841901 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:50.841976 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 12:17:50.842307 9267 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260620 12:17:50.842504 9375 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:17:50.842916 9267 tablet_bootstrap.cc:492] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Bootstrap complete.
I20260620 12:17:50.844000 9267 ts_tablet_manager.cc:1397] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Time spent bootstrapping tablet: real 0.046s user 0.029s sys 0.009s
I20260620 12:17:50.846917 9267 raft_consensus.cc:348] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:50.847527 9267 raft_consensus.cc:729] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71aa47f95254554a6cfa10a8117ccd3, State: Initialized, Role: FOLLOWER
I20260620 12:17:50.848107 9267 consensus_queue.cc:260] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:50.849357 9267 ts_tablet_manager.cc:1428] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 12:17:50.849896 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:50.849546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:52795)
0620 12:17:50.849723 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:50.849811 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:17:50.850867 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:50.850525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:42803)
0620 12:17:50.850700 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:50.850784 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 12:17:50.974215 9378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:50.974637 9378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:50.974691 9378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:50.979543 9378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:50.979667 9378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:50.979743 9378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:17:50.986730 9378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:33401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=46071
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:50.988394 9378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:50.989877 9378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:51.004606 9385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:51.004612 9386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:51.004702 9378 server_base.cc:1034] running on GCE node
W20260620 12:17:51.004612 9388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:51.005751 9378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:51.006739 9378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:51.008092 9378 hybrid_clock.cc:648] HybridClock initialized: now 1781957871008045 us; error 73 us; skew 500 ppm
Jun 20 12:17:51 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957871, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:51.012113 9378 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:17:51.013666 9378 webserver.cc:460] Webserver started at http://127.24.141.2:46071/ using document root <none> and password file <none>
I20260620 12:17:51.014482 9378 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:51.014575 9378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:51.020351 9378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:17:51.023695 9395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:51.025609 9378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:17:51.025813 9378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "2869af5c64024e0db4e0eaeffe13df35"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "84c705ef54287eba044959fec5e781d4"
server_key_iv: "7633cd90b5f43b6f8fccb4a577766588"
server_key_version: "encryptionkey@0"
I20260620 12:17:51.026594 9378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:51.047344 9378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:51.051645 9378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:51.051911 9378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:51.052788 9378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:51.054003 9378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:17:51.054064 9378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:51.054129 9378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:17:51.054155 9378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:51.067741 9378 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:33401
I20260620 12:17:51.067818 9508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:33401 every 8 connection(s)
I20260620 12:17:51.069371 9378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:17:51.073520 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 9378
I20260620 12:17:51.074093 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:36239
--local_ip_for_outbound_sockets=127.24.141.3
--tserver_master_addrs=127.24.141.62:40207
--webserver_port=42895
--webserver_interface=127.24.141.3
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:17:51.075423 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:51.072625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:46757)
0620 12:17:51.073579 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:51.073833 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":183,"threads_started":1}
W20260620 12:17:51.076040 9509 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:40207 (0 consecutive failures): Network error: Failed to ping master at 127.24.141.62:40207: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
I20260620 12:17:51.076858 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:51.076576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:36761)
0620 12:17:51.076715 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:51.076784 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 12:17:51.077587 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:51.077368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:46439)
0620 12:17:51.077480 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:51.077530 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 12:17:51.077767 9509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:17:51.215420 9512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:17:51.215921 9512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:17:51.215994 9512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:17:51.221313 9512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:17:51.221467 9512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:17:51.221573 9512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:17:51.228531 9512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:36239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=42895
--tserver_master_addrs=127.24.141.62:40207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:17:51.230201 9512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:17:51.231688 9512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:17:51.244477 9517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:51.244477 9518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:17:51.244517 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:17:51.244665 9512 server_base.cc:1034] running on GCE node
I20260620 12:17:51.245539 9512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:17:51.246495 9512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:17:51.247709 9512 hybrid_clock.cc:648] HybridClock initialized: now 1781957871247689 us; error 54 us; skew 500 ppm
Jun 20 12:17:51 dist-test-slave-rh5v krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957871, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:17:51.251397 9512 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:17:51.252777 9512 webserver.cc:460] Webserver started at http://127.24.141.3:42895/ using document root <none> and password file <none>
I20260620 12:17:51.253675 9512 fs_manager.cc:359] Metadata directory not provided
I20260620 12:17:51.253762 9512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:17:51.259724 9512 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:17:51.263347 9527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:51.266079 9512 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 12:17:51.266355 9512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "1927ab05b8d44dfaba4c061e8c4ab049"
format_stamp: "Formatted at 2026-06-20 12:17:08 on dist-test-slave-rh5v"
server_key: "67b7f5eb6b052cf1c56ebf81c640fa48"
server_key_iv: "f0261b984e0f0f45ffeb9decf2e1b574"
server_key_version: "encryptionkey@0"
I20260620 12:17:51.267277 9512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:17:51.291222 9512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:17:51.296432 9512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:17:51.296736 9512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:17:51.297670 9512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:17:51.298965 9512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:17:51.299036 9512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:51.299113 9512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:17:51.299144 9512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:17:51.311784 9512 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:36239
I20260620 12:17:51.312004 9640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:36239 every 8 connection(s)
I20260620 12:17:51.313589 9512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:17:51.315147 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 9512
I20260620 12:17:51.319607 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:51.317161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:38361)
0620 12:17:51.318486 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:51.318677 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":969,"thread_start_us":670,"threads_started":1}
W20260620 12:17:51.320469 9641 heartbeater.cc:644] Failed to heartbeat to 127.24.141.62:40207 (0 consecutive failures): Network error: Failed to ping master at 127.24.141.62:40207: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
I20260620 12:17:51.321591 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:51.321199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:48075)
0620 12:17:51.321421 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:51.321507 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 12:17:51.322736 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:51.322337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:56321)
0620 12:17:51.322557 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:51.322630 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 12:17:51.323181 9641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:17:51.852661 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:51.851989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:59953)
0620 12:17:51.852420 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:51.852519 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":112,"threads_started":1}
I20260620 12:17:52.070854 9646 raft_consensus.cc:482] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:17:52.071120 9646 raft_consensus.cc:504] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:52.072547 9646 leader_election.cc:288] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401), 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
I20260620 12:17:52.079306 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:52.078596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:37289)
0620 12:17:52.079029 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:52.079129 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":147,"threads_started":1}
Jun 20 12:17:52 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957870, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.2@KRBTEST.COM
Jun 20 12:17:52 dist-test-slave-rh5v krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957870, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.3@KRBTEST.COM
I20260620 12:17:52.093300 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:52.072918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36239 (local address 127.24.141.1:32825)
0620 12:17:52.073516 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:52.073534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:52.073677 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:52.076009 (+ 2332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:52.076013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:52.076530 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:52.077194 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:52.077222 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.078937 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.078942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:52.080399 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:52.080418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.081562 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.081567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:52.081679 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:52.082527 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:52.082558 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:52.085529 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:52.088927 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:52.088931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:52.088934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:52.089216 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:52.089826 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:52.089836 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:52.089838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:52.090165 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:52.091705 (+ 1540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:52.091715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:52.092207 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 12:17:52.092495 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 12:17:52.092990 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":137,"spinlock_wait_cycles":29056,"thread_start_us":149,"threads_started":1}
I20260620 12:17:52.093300 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:52.072756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:33401 (local address 127.24.141.1:38007)
0620 12:17:52.072951 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:52.072969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:17:52.073137 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:17:52.075883 (+ 2746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:17:52.075890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:17:52.076591 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:17:52.077196 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:52.077223 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.079083 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.079087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:52.080402 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:17:52.080421 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.081441 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.081445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:17:52.081662 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:52.082539 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:17:52.082552 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:17:52.085392 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:17:52.088598 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:52.088607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:52.088625 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:52.089039 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:52.089826 (+ 787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:17:52.089836 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:17:52.089838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:17:52.090165 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:17:52.091705 (+ 1540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:17:52.091715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:17:52.092207 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 12:17:52.092495 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 12:17:52.093104 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":429,"spinlock_wait_cycles":34688}
I20260620 12:17:52.093780 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:52.073097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:38007 (local address 127.24.141.2:33401)
0620 12:17:52.073558 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:52.073585 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:52.073638 (+ 53us) server_negotiation.cc:407] Connection header received
0620 12:17:52.074674 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:52.074696 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:52.075288 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:52.075697 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:52.077457 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.078836 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.080595 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.081270 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.081481 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:52.085867 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:52.085875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:52.085892 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:52.085938 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:52.088363 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:52.089337 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:52.089349 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:52.089359 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:52.089666 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:52.090375 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:52.090380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:52.090382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:52.091494 (+ 1112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:52.091701 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:52.092792 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0620 12:17:52.093433 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":85,"threads_started":1}
I20260620 12:17:52.094374 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:52.073337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:32825 (local address 127.24.141.3:36239)
0620 12:17:52.073913 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 12:17:52.073930 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:17:52.074006 (+ 76us) server_negotiation.cc:407] Connection header received
0620 12:17:52.075145 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:17:52.075154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:17:52.075623 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:17:52.076011 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:17:52.077443 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.078710 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.080607 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:17:52.081360 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:17:52.081570 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:17:52.085849 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:17:52.085858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:17:52.085879 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:17:52.085926 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:17:52.088769 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:52.089392 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:52.089398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:52.089403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:52.089666 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:17:52.090374 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:17:52.090380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:17:52.090382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:17:52.091494 (+ 1112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:17:52.091701 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 12:17:52.092792 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0620 12:17:52.094199 (+ 1407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":136,"threads_started":1}
W20260620 12:17:52.097249 9262 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
W20260620 12:17:52.097517 9264 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:17:52.097682 9264 leader_election.cc:302] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71aa47f95254554a6cfa10a8117ccd3; no voters: 1927ab05b8d44dfaba4c061e8c4ab049, 2869af5c64024e0db4e0eaeffe13df35
I20260620 12:17:52.098037 9646 raft_consensus.cc:2726] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:17:52.324980 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:52.324276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:36467)
0620 12:17:52.324709 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:52.324831 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
I20260620 12:17:52.854761 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:52.853970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:45003)
0620 12:17:52.854482 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:52.854574 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":114,"threads_started":1}
I20260620 12:17:53.081550 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:53.080745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:50535)
0620 12:17:53.081284 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:53.081383 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":134,"threads_started":1}
I20260620 12:17:53.327175 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:53.326442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:40547)
0620 12:17:53.326919 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:53.327025 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":116,"threads_started":1}
I20260620 12:17:53.857190 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:53.856227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:39455)
0620 12:17:53.856862 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:53.856970 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":321,"threads_started":1}
I20260620 12:17:53.988548 9656 raft_consensus.cc:482] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:17:53.988760 9656 raft_consensus.cc:504] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:53.989349 9656 leader_election.cc:288] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401), 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
W20260620 12:17:53.990406 9264 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
W20260620 12:17:53.990689 9262 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:17:53.990803 9262 leader_election.cc:302] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71aa47f95254554a6cfa10a8117ccd3; no voters: 1927ab05b8d44dfaba4c061e8c4ab049, 2869af5c64024e0db4e0eaeffe13df35
I20260620 12:17:53.991044 9656 raft_consensus.cc:2726] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:17:54.083745 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:54.082885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:36089)
0620 12:17:54.083438 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:54.083546 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":118,"threads_started":1}
I20260620 12:17:54.329017 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:54.328407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:44541)
0620 12:17:54.328793 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:54.328868 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":110,"threads_started":1}
I20260620 12:17:54.859031 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:54.858420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:50773)
0620 12:17:54.858834 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:54.858913 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260620 12:17:55.085538 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:55.084819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:36281)
0620 12:17:55.085293 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:55.085392 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":98,"threads_started":1}
I20260620 12:17:55.330638 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:55.329988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:51431)
0620 12:17:55.330351 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:55.330478 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":83,"threads_started":1}
I20260620 12:17:55.617641 9662 raft_consensus.cc:482] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:17:55.617856 9662 raft_consensus.cc:504] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:55.618465 9662 leader_election.cc:288] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401), 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
W20260620 12:17:55.619652 9262 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
W20260620 12:17:55.619787 9264 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:17:55.619930 9264 leader_election.cc:302] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71aa47f95254554a6cfa10a8117ccd3; no voters: 1927ab05b8d44dfaba4c061e8c4ab049, 2869af5c64024e0db4e0eaeffe13df35
I20260620 12:17:55.620184 9662 raft_consensus.cc:2726] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:17:55.860812 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:55.860204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:60215)
0620 12:17:55.860569 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:55.860675 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260620 12:17:56.087644 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:56.087033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:48667)
0620 12:17:56.087434 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:56.087513 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260620 12:17:56.332325 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:56.331765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:59611)
0620 12:17:56.332100 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:56.332184 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":213,"threads_started":1}
I20260620 12:17:56.862706 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:56.862032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:53809)
0620 12:17:56.862442 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:56.862557 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260620 12:17:57.089350 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:57.088647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:40525)
0620 12:17:57.089051 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:57.089197 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":126,"threads_started":1}
I20260620 12:17:57.334112 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:57.333486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:38175)
0620 12:17:57.333863 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:57.333961 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
I20260620 12:17:57.864228 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:57.863703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:51581)
0620 12:17:57.864013 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:57.864091 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260620 12:17:58.091060 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:58.090458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:38367)
0620 12:17:58.090855 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:58.090925 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":101,"threads_started":1}
I20260620 12:17:58.335935 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:58.335246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:34855)
0620 12:17:58.335695 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:58.335785 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":121,"threads_started":1}
I20260620 12:17:58.866025 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:58.865399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:45443)
0620 12:17:58.865794 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:58.865887 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":103,"threads_started":1}
I20260620 12:17:59.092846 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:59.092272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:53725)
0620 12:17:59.092627 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:59.092705 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":128,"threads_started":1}
I20260620 12:17:59.337827 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:59.337143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:54153)
0620 12:17:59.337592 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:59.337689 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":123,"threads_started":1}
I20260620 12:17:59.492467 9675 raft_consensus.cc:482] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:17:59.492609 9675 raft_consensus.cc:504] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:17:59.493137 9675 leader_election.cc:288] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401), 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
W20260620 12:17:59.494236 9262 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
W20260620 12:17:59.494385 9264 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:17:59.494536 9264 leader_election.cc:302] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71aa47f95254554a6cfa10a8117ccd3; no voters: 1927ab05b8d44dfaba4c061e8c4ab049, 2869af5c64024e0db4e0eaeffe13df35
I20260620 12:17:59.494736 9675 raft_consensus.cc:2726] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:17:59.867615 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:17:59.867036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:43677)
0620 12:17:59.867389 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 12:17:59.867474 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260620 12:18:00.094807 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:00.094101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:46637)
0620 12:18:00.094554 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:00.094658 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
I20260620 12:18:00.339531 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:00.338888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:34429)
0620 12:18:00.339283 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:00.339381 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
I20260620 12:18:00.869817 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:00.869136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:34795)
0620 12:18:00.869578 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:00.869673 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260620 12:18:01.096622 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:01.095969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:50361)
0620 12:18:01.096406 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:01.096499 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":110,"threads_started":1}
I20260620 12:18:01.341361 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:01.340687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:56431)
0620 12:18:01.341115 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:01.341220 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
I20260620 12:18:01.871605 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:01.871003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:49261)
0620 12:18:01.871384 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:01.871467 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":114,"threads_started":1}
I20260620 12:18:02.098471 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:02.097755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:45029)
0620 12:18:02.098257 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:02.098350 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":126,"threads_started":1}
I20260620 12:18:02.343218 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:02.342472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:59357)
0620 12:18:02.342855 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:02.343010 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":90,"threads_started":1}
I20260620 12:18:02.873375 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:02.872782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:56503)
0620 12:18:02.873159 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:02.873240 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260620 12:18:03.100345 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:03.099704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:45175)
0620 12:18:03.100122 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:03.100218 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":110,"threads_started":1}
I20260620 12:18:03.344710 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:03.344214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:55481)
0620 12:18:03.344542 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:03.344606 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
I20260620 12:18:03.875219 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:03.874594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:42883)
0620 12:18:03.874994 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:03.875084 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260620 12:18:04.102109 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:04.101373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:35927)
0620 12:18:04.101849 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:04.101954 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":114,"threads_started":1}
I20260620 12:18:04.346562 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:04.345910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:49673)
0620 12:18:04.346282 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:04.346364 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260620 12:18:04.876768 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:04.876107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:34151)
0620 12:18:04.876502 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:04.876623 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":181,"threads_started":1}
I20260620 12:18:05.041170 9692 raft_consensus.cc:482] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:18:05.041363 9692 raft_consensus.cc:504] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2869af5c64024e0db4e0eaeffe13df35" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 33401 } } peers { permanent_uuid: "1927ab05b8d44dfaba4c061e8c4ab049" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36239 } } peers { permanent_uuid: "c71aa47f95254554a6cfa10a8117ccd3" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 38519 } }
I20260620 12:18:05.041934 9692 leader_election.cc:288] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401), 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239)
W20260620 12:18:05.043108 9262 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1927ab05b8d44dfaba4c061e8c4ab049 (127.24.141.3:36239): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
W20260620 12:18:05.043263 9264 leader_election.cc:341] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2869af5c64024e0db4e0eaeffe13df35 (127.24.141.2:33401): Not found: Tablet not found: 6eda1fd6ec3146f4b6a7ba6d5eb0957f
I20260620 12:18:05.043408 9264 leader_election.cc:302] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71aa47f95254554a6cfa10a8117ccd3; no voters: 1927ab05b8d44dfaba4c061e8c4ab049, 2869af5c64024e0db4e0eaeffe13df35
I20260620 12:18:05.043702 9692 raft_consensus.cc:2726] T 6eda1fd6ec3146f4b6a7ba6d5eb0957f P c71aa47f95254554a6cfa10a8117ccd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:18:05.103833 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:05.103224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:33777)
0620 12:18:05.103613 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:05.103703 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":101,"threads_started":1}
I20260620 12:18:05.348373 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:05.347728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.3:49979)
0620 12:18:05.348153 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:05.348244 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":103,"threads_started":1}
I20260620 12:18:05.878566 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:05.878044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.1:38507)
0620 12:18:05.878357 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:05.878426 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260620 12:18:06.105677 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:06.105140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40207 (local address 127.24.141.2:46815)
0620 12:18:06.105454 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:06.105538 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.62:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260620 12:18:06.322671 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 9205
I20260620 12:18:06.330783 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 9378
I20260620 12:18:06.339344 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 9512
2026-06-20T12:18:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59890 ms)
[----------] 30 tests from SecurityITest (267582 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:06 dist-test-slave-rh5v krb5kdc[9699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:06 dist-test-slave-rh5v krb5kdc[9699](info): set up 2 sockets
Jun 20 12:18:06 dist-test-slave-rh5v krb5kdc[9699](info): commencing operation
krb5kdc: starting...
W20260620 12:18:07.333529 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:07 dist-test-slave-rh5v krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:07.475852 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40819
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:40819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:07.608736 9712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:07.609093 9712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:07.609151 9712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:07.613547 9712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:07.613641 9712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:07.613660 9712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:07.613677 9712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:07.613692 9712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:07.619694 9712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:40819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:07.621004 9712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:07.622231 9712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:07.633646 9717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:07.633704 9720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:07.633728 9718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:07.634027 9712 server_base.cc:1034] running on GCE node
I20260620 12:18:07.634727 9712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:07.636046 9712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:07.637280 9712 hybrid_clock.cc:648] HybridClock initialized: now 1781957887637255 us; error 50 us; skew 500 ppm
Jun 20 12:18:07 dist-test-slave-rh5v krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957887, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:07.644445 9712 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:07.645794 9712 webserver.cc:460] Webserver started at http://127.24.141.62:34965/ using document root <none> and password file <none>
I20260620 12:18:07.646503 9712 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:07.646587 9712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:07.646845 9712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:07.649334 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "489901dfe4fa4f33a9180c64a149d20d"
format_stamp: "Formatted at 2026-06-20 12:18:07 on dist-test-slave-rh5v"
server_key: "5b7a4f33a87ecc83cb790a63e3e9bed9"
server_key_iv: "cedd6ac4270e9ba235ce6e21ea6b6c57"
server_key_version: "encryptionkey@0"
I20260620 12:18:07.649992 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "489901dfe4fa4f33a9180c64a149d20d"
format_stamp: "Formatted at 2026-06-20 12:18:07 on dist-test-slave-rh5v"
server_key: "5b7a4f33a87ecc83cb790a63e3e9bed9"
server_key_iv: "cedd6ac4270e9ba235ce6e21ea6b6c57"
server_key_version: "encryptionkey@0"
I20260620 12:18:07.655141 9712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 12:18:07.659446 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:07.661409 9712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 12:18:07.661604 9712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "489901dfe4fa4f33a9180c64a149d20d"
format_stamp: "Formatted at 2026-06-20 12:18:07 on dist-test-slave-rh5v"
server_key: "5b7a4f33a87ecc83cb790a63e3e9bed9"
server_key_iv: "cedd6ac4270e9ba235ce6e21ea6b6c57"
server_key_version: "encryptionkey@0"
I20260620 12:18:07.661784 9712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:07.679569 9712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:07.683317 9712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:07.683548 9712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:07.691915 9712 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:40819
I20260620 12:18:07.691954 9779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:40819 every 8 connection(s)
I20260620 12:18:07.693156 9712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:07.695674 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 9712
I20260620 12:18:07.695863 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:07.696255 25140 external_mini_cluster.cc:1428] Setting key 715065198254e6a9e1532049c9c394f3
I20260620 12:18:07.697613 9780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:07.706200 9780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d: Bootstrap starting.
Jun 20 12:18:07 dist-test-slave-rh5v krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:07.709537 9780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:07.710428 9780 log.cc:824] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:07.713436 9780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d: No bootstrap required, opened a new log
I20260620 12:18:07.714201 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:07.698251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.24.141.62:40819)
0620 12:18:07.698794 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:07.698812 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:07.698857 (+ 45us) server_negotiation.cc:407] Connection header received
0620 12:18:07.699597 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:07.699616 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:07.700013 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:07.700502 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:07.701644 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:07.702805 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:07.704179 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:07.704831 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:07.705006 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:07.708467 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:07.708475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:07.708496 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:07.708547 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:07.710536 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:07.711185 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:07.711193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:07.711201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:07.711397 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:07.711725 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:07.711728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:07.711729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:07.712241 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:07.712423 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:07.712807 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 12:18:07.713238 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":150,"threads_started":1}
I20260620 12:18:07.717623 9780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489901dfe4fa4f33a9180c64a149d20d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40819 } }
I20260620 12:18:07.717921 9780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:07.717980 9780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489901dfe4fa4f33a9180c64a149d20d, State: Initialized, Role: FOLLOWER
I20260620 12:18:07.718506 9780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489901dfe4fa4f33a9180c64a149d20d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40819 } }
I20260620 12:18:07.718665 9780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:07.718735 9780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:07.718844 9780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:07.720043 9780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489901dfe4fa4f33a9180c64a149d20d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40819 } }
I20260620 12:18:07.720501 9780 leader_election.cc:302] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 489901dfe4fa4f33a9180c64a149d20d; no voters:
I20260620 12:18:07.720896 9780 leader_election.cc:288] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:07.721019 9785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:07.721284 9785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [term 1 LEADER]: Becoming Leader. State: Replica: 489901dfe4fa4f33a9180c64a149d20d, State: Running, Role: LEADER
I20260620 12:18:07.721670 9785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489901dfe4fa4f33a9180c64a149d20d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40819 } }
I20260620 12:18:07.722585 9780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:07.723773 9787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 489901dfe4fa4f33a9180c64a149d20d. Latest consensus state: current_term: 1 leader_uuid: "489901dfe4fa4f33a9180c64a149d20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489901dfe4fa4f33a9180c64a149d20d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40819 } } }
I20260620 12:18:07.724009 9787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:07.724557 9794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:07.724515 9786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "489901dfe4fa4f33a9180c64a149d20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489901dfe4fa4f33a9180c64a149d20d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40819 } } }
I20260620 12:18:07.724714 9786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:07.728834 9794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:07.736927 9794 catalog_manager.cc:1347] Generated new cluster ID: 3dd716f357c4484ba238bc13d8fb0a22
I20260620 12:18:07.737116 9794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:07.763175 9794 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:07.764554 9794 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:07.780234 9794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d: Generated new TSK 0
I20260620 12:18:07.781162 9794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:18:07.820549 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40819
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:07.951745 9806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:07.952112 9806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:07.952181 9806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:07.958345 9806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:07.958472 9806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:07.958596 9806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:18:07.965425 9806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:07.966741 9806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:07.967944 9806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:07.979065 9814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:07.979076 9811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:07.979076 9812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:07.979913 9806 server_base.cc:1034] running on GCE node
I20260620 12:18:07.981117 9806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:07.982033 9806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:07.983259 9806 hybrid_clock.cc:648] HybridClock initialized: now 1781957887983234 us; error 43 us; skew 500 ppm
Jun 20 12:18:07 dist-test-slave-rh5v krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957887, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:07.987169 9806 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:18:07.988550 9806 webserver.cc:460] Webserver started at http://127.24.141.1:45383/ using document root <none> and password file <none>
I20260620 12:18:07.989542 9806 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:07.989647 9806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:07.989970 9806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:07.993498 9806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "cc139f2054204a3fb739bd5cfe1e5ff5"
format_stamp: "Formatted at 2026-06-20 12:18:07 on dist-test-slave-rh5v"
server_key: "11858e426776310a5021306e3e2e04d9"
server_key_iv: "0f66da7c00e0e57ac0600bad456db029"
server_key_version: "encryptionkey@0"
I20260620 12:18:07.994357 9806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "cc139f2054204a3fb739bd5cfe1e5ff5"
format_stamp: "Formatted at 2026-06-20 12:18:07 on dist-test-slave-rh5v"
server_key: "11858e426776310a5021306e3e2e04d9"
server_key_iv: "0f66da7c00e0e57ac0600bad456db029"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.000758 9806 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 12:18:08.004639 9821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.006537 9806 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:08.006722 9806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "cc139f2054204a3fb739bd5cfe1e5ff5"
format_stamp: "Formatted at 2026-06-20 12:18:07 on dist-test-slave-rh5v"
server_key: "11858e426776310a5021306e3e2e04d9"
server_key_iv: "0f66da7c00e0e57ac0600bad456db029"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.006873 9806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:08.025672 9806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:08.030078 9806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:08.030320 9806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:08.031136 9806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:08.032280 9806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:08.032341 9806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.032411 9806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:08.032438 9806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.045655 9806 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:42531
I20260620 12:18:08.045715 9934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:42531 every 8 connection(s)
I20260620 12:18:08.046893 9806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:18:08.051354 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 9806
I20260620 12:18:08.051510 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:18:08.051834 25140 external_mini_cluster.cc:1428] Setting key 3bafa4684d5c1b207a0b1a4414042ef3
Jun 20 12:18:08 dist-test-slave-rh5v krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957887, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:18:08.064687 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.049449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:51867 (local address 127.24.141.62:40819)
0620 12:18:08.049631 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.049636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.050976 (+ 1340us) server_negotiation.cc:407] Connection header received
0620 12:18:08.052760 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.052764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.052836 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.052917 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:08.054567 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.055127 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.056388 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.056634 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.056728 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.059727 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:08.059732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:08.059734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:08.059777 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:08.061677 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.062562 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.062565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.062567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.062682 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.063244 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.063247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.063249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.063511 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:08.063610 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.064328 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.064498 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:18:08.065821 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.050008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40819 (local address 127.24.141.1:51867)
0620 12:18:08.050746 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.050820 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.052465 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.053203 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.053212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.053752 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:08.054388 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.054406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.055275 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.055278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.056248 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.056260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.056808 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.056812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.056993 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.057743 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:08.057763 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:08.059434 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:08.061881 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:08.061890 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:08.061916 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:08.062401 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:08.062802 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:08.062805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:08.062807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:08.063040 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:08.063630 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:08.063637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:08.064048 (+ 411us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.064253 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.064576 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":143,"threads_started":1}
I20260620 12:18:08.067826 9935 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40819
I20260620 12:18:08.068221 9935 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:08.069181 9935 heartbeater.cc:511] Master 127.24.141.62:40819 requested a full tablet report, sending...
I20260620 12:18:08.071583 9742 ts_manager.cc:194] Registered new tserver with Master: cc139f2054204a3fb739bd5cfe1e5ff5 (127.24.141.1:42531)
I20260620 12:18:08.073453 9742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:51867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:18:08.085312 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40819
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:08.219436 9940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:08.219833 9940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:08.219885 9940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:08.224684 9940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:08.224789 9940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:08.224874 9940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:18:08.231552 9940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:08.233171 9940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:08.234536 9940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:08.246897 9945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:08.246897 9948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:08.246897 9946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:08.247329 9940 server_base.cc:1034] running on GCE node
I20260620 12:18:08.248035 9940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:08.248919 9940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:08.250115 9940 hybrid_clock.cc:648] HybridClock initialized: now 1781957888250102 us; error 45 us; skew 500 ppm
Jun 20 12:18:08 dist-test-slave-rh5v krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957888, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:08.253614 9940 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:18:08.254777 9940 webserver.cc:460] Webserver started at http://127.24.141.2:39013/ using document root <none> and password file <none>
I20260620 12:18:08.255486 9940 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:08.255563 9940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:08.255802 9940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:08.258612 9940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "a405b2056cff4187850eb418ce6624af"
format_stamp: "Formatted at 2026-06-20 12:18:08 on dist-test-slave-rh5v"
server_key: "34a3f46c345fbefd5c7a34d613d6fbde"
server_key_iv: "0978f70fc7f9b69d9ae12d776719afef"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.259344 9940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "a405b2056cff4187850eb418ce6624af"
format_stamp: "Formatted at 2026-06-20 12:18:08 on dist-test-slave-rh5v"
server_key: "34a3f46c345fbefd5c7a34d613d6fbde"
server_key_iv: "0978f70fc7f9b69d9ae12d776719afef"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.266041 9940 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 12:18:08.270641 9955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.272804 9940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:18:08.273038 9940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "a405b2056cff4187850eb418ce6624af"
format_stamp: "Formatted at 2026-06-20 12:18:08 on dist-test-slave-rh5v"
server_key: "34a3f46c345fbefd5c7a34d613d6fbde"
server_key_iv: "0978f70fc7f9b69d9ae12d776719afef"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.273301 9940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:08.286265 9940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:08.290836 9940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:08.291060 9940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:08.291811 9940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:08.292842 9940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:08.292901 9940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.292958 9940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:08.292981 9940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.304649 9940 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:42889
I20260620 12:18:08.304715 10068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:42889 every 8 connection(s)
I20260620 12:18:08.305887 9940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:18:08.314594 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 9940
I20260620 12:18:08.314747 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:18:08.315068 25140 external_mini_cluster.cc:1428] Setting key 1e89de461e7594d776501efc39fcd1f4
Jun 20 12:18:08 dist-test-slave-rh5v krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957888, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:08.323577 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.308293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:59441 (local address 127.24.141.62:40819)
0620 12:18:08.308495 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.308500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.309744 (+ 1244us) server_negotiation.cc:407] Connection header received
0620 12:18:08.311311 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.311314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.311387 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.311469 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:08.312974 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.313588 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.314857 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.315089 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.315199 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.318818 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:08.318821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:08.318824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:08.318855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:08.320308 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.321175 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.321180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.321182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.321328 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.321845 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.321848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.321849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.322121 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:08.322261 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.323133 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.323340 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:18:08.324791 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.308783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40819 (local address 127.24.141.2:59441)
0620 12:18:08.309534 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.309602 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.311019 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.311707 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.311715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.312153 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:08.312796 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.312815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.313725 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.313729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.314680 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.314694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.315360 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.315368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.315686 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.316458 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:08.316481 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:08.318582 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:08.320486 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:08.320495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:08.320518 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:08.320940 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:08.321468 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:08.321472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:08.321474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:08.321705 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:08.322305 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:08.322314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:08.322800 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.323060 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.323621 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":178,"threads_started":1}
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
I20260620 12:18:08.326546 10069 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40819
I20260620 12:18:08.326967 10069 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:08.327665 10069 heartbeater.cc:511] Master 127.24.141.62:40819 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:18:08.329257 9742 ts_manager.cc:194] Registered new tserver with Master: a405b2056cff4187850eb418ce6624af (127.24.141.2:42889)
I20260620 12:18:08.329957 9742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:59441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:18:08.348496 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40819
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:08.480686 10074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:08.481040 10074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:08.481141 10074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:08.485589 10074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:08.485682 10074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:08.485746 10074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:18:08.491865 10074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:08.493368 10074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:08.494706 10074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:08.507364 10082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:08.507331 10080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:08.507390 10074 server_base.cc:1034] running on GCE node
W20260620 12:18:08.507325 10079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:08.508345 10074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:08.509411 10074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:08.510710 10074 hybrid_clock.cc:648] HybridClock initialized: now 1781957888510700 us; error 58 us; skew 500 ppm
Jun 20 12:18:08 dist-test-slave-rh5v krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957888, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:08.515085 10074 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:18:08.516634 10074 webserver.cc:460] Webserver started at http://127.24.141.3:32995/ using document root <none> and password file <none>
I20260620 12:18:08.517534 10074 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:08.517643 10074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:08.517930 10074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:08.520816 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "5c15adf072434c9daba9e5197d7bc675"
format_stamp: "Formatted at 2026-06-20 12:18:08 on dist-test-slave-rh5v"
server_key: "cf2e46f2f52583820013de9f69a87165"
server_key_iv: "3eb95a193560b13018ac391c066e2bb5"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.521663 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "5c15adf072434c9daba9e5197d7bc675"
format_stamp: "Formatted at 2026-06-20 12:18:08 on dist-test-slave-rh5v"
server_key: "cf2e46f2f52583820013de9f69a87165"
server_key_iv: "3eb95a193560b13018ac391c066e2bb5"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.527139 10074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:08.531352 10089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.533622 10074 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:08.533834 10074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "5c15adf072434c9daba9e5197d7bc675"
format_stamp: "Formatted at 2026-06-20 12:18:08 on dist-test-slave-rh5v"
server_key: "cf2e46f2f52583820013de9f69a87165"
server_key_iv: "3eb95a193560b13018ac391c066e2bb5"
server_key_version: "encryptionkey@0"
I20260620 12:18:08.533995 10074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:08.547567 10074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:08.551921 10074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:08.552170 10074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:08.552997 10074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:08.554113 10074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:08.554167 10074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.554229 10074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:08.554255 10074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:08.567026 10074 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:37257
I20260620 12:18:08.567094 10202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:37257 every 8 connection(s)
I20260620 12:18:08.568468 10074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:18:08.578053 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 10074
I20260620 12:18:08.578213 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:18:08.578567 25140 external_mini_cluster.cc:1428] Setting key e5046cd8df0fa9a82a39f4b543825b4f
Jun 20 12:18:08 dist-test-slave-rh5v krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957888, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:08.587087 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.571060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:50323 (local address 127.24.141.62:40819)
0620 12:18:08.571308 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.571315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.572641 (+ 1326us) server_negotiation.cc:407] Connection header received
0620 12:18:08.574233 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.574237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.574315 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.574423 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:08.575949 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.576553 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.578598 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.578959 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.579092 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.582424 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:08.582428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:08.582430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:08.582467 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:08.584282 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.585239 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.585243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.585245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.585375 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.585778 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.585781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.585782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.586011 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:08.586107 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.586712 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.586896 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 12:18:08.588075 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.571673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40819 (local address 127.24.141.3:50323)
0620 12:18:08.572404 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.572475 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.573949 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.574629 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.574638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.575109 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:08.575732 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.575756 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.576765 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.576770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.578377 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.578400 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.579212 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.579218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.579489 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.580360 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:08.580385 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:08.582198 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:08.584558 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:08.584567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:08.584587 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:08.585019 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:08.585484 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:08.585487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:08.585489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:08.585666 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:08.586121 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:08.586130 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:08.586439 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.586626 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.587034 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":160,"threads_started":1}
I20260620 12:18:08.590157 10203 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40819
I20260620 12:18:08.590525 10203 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:08.591192 10203 heartbeater.cc:511] Master 127.24.141.62:40819 requested a full tablet report, sending...
I20260620 12:18:08.592831 9742 ts_manager.cc:194] Registered new tserver with Master: 5c15adf072434c9daba9e5197d7bc675 (127.24.141.3:37257)
I20260620 12:18:08.593817 9742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:50323
I20260620 12:18:08.594414 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:18:08.607311 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.597422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.24.141.62:40819)
0620 12:18:08.597564 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.597568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.597785 (+ 217us) server_negotiation.cc:407] Connection header received
0620 12:18:08.597872 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.597875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.597949 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.598052 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:08.598850 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.599449 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.600549 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.600928 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.601121 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.603023 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:08.603027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:08.603034 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:08.603080 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:08.604939 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.605622 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.605628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.605629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.605820 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:08.606184 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:08.606187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:08.606189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:08.606488 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:08.606615 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.607001 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.607174 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:18:08.611728 9742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:18:08.616281 9742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:18:08.636215 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.629919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:37257 (local address 127.0.0.1:54268)
0620 12:18:08.630334 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.630352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.630620 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.631648 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.631656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.631684 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:08.631911 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.631927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.633206 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.633210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.634495 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.634505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.635451 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.635455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.635614 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.635672 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.635820 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.636010 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":128,"threads_started":1}
I20260620 12:18:08.636346 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.630177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.24.141.3:37257)
0620 12:18:08.630940 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.630952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.630981 (+ 29us) server_negotiation.cc:407] Connection header received
0620 12:18:08.631085 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.631091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.631392 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.631626 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:08.632080 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.633037 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.634640 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.635309 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.635429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.635615 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.635876 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.636007 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":188,"threads_started":1}
I20260620 12:18:08.637152 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.629219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:42889 (local address 127.0.0.1:49038)
0620 12:18:08.630064 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.630143 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.630318 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.631856 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.631859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.631874 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:08.632058 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.632066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.633752 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.633755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.635204 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.635211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.636665 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.636669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.636780 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.636806 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.636883 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.636973 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":104,"thread_start_us":249,"threads_started":1}
I20260620 12:18:08.637287 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.629235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.24.141.2:42889)
0620 12:18:08.630974 (+ 1739us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.630988 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.631018 (+ 30us) server_negotiation.cc:407] Connection header received
0620 12:18:08.631110 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.631116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.631394 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.631644 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:08.632202 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.633570 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.635380 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.636288 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.636404 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.636599 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.636949 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.637127 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1608,"thread_start_us":227,"threads_started":1}
I20260620 12:18:08.638029 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.630950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.24.141.1:42531)
0620 12:18:08.631447 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.631455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.631773 (+ 318us) server_negotiation.cc:407] Connection header received
0620 12:18:08.632043 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.632050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.632249 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.632407 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:08.632656 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.633789 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.635379 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.636420 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.636598 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.636753 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.637703 (+ 950us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.637845 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":140,"threads_started":1}
I20260620 12:18:08.637596 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.630318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:42531 (local address 127.0.0.1:37744)
0620 12:18:08.631659 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.631675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.631855 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.632407 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.632410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.632431 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:08.632547 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.632553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.633917 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.633920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.635194 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.635207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.636819 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.636822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.636928 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.636952 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.637022 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.637134 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":10,"thread_start_us":106,"threads_started":1}
I20260620 12:18:08.641132 10003 tablet_service.cc:1467] Processing CreateTablet for tablet bf3899ce0b8e488593c10d8b14b3356e (DEFAULT_TABLE table=test-table [id=8490e738aa0e48c8ba8f88701fdafbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:08.641255 10137 tablet_service.cc:1467] Processing CreateTablet for tablet bf3899ce0b8e488593c10d8b14b3356e (DEFAULT_TABLE table=test-table [id=8490e738aa0e48c8ba8f88701fdafbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:08.642707 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3899ce0b8e488593c10d8b14b3356e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:08.642864 10137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3899ce0b8e488593c10d8b14b3356e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:08.644356 9869 tablet_service.cc:1467] Processing CreateTablet for tablet bf3899ce0b8e488593c10d8b14b3356e (DEFAULT_TABLE table=test-table [id=8490e738aa0e48c8ba8f88701fdafbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:08.645957 9869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3899ce0b8e488593c10d8b14b3356e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:08.650874 10220 tablet_bootstrap.cc:492] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Bootstrap starting.
I20260620 12:18:08.653980 10221 tablet_bootstrap.cc:492] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Bootstrap starting.
I20260620 12:18:08.654071 10220 tablet_bootstrap.cc:654] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:08.655550 10220 log.cc:824] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:08.656488 10222 tablet_bootstrap.cc:492] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5: Bootstrap starting.
I20260620 12:18:08.657369 10221 tablet_bootstrap.cc:654] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:08.658300 10220 tablet_bootstrap.cc:492] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: No bootstrap required, opened a new log
I20260620 12:18:08.658562 10220 ts_tablet_manager.cc:1397] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:18:08.658957 10221 log.cc:824] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:08.659586 10222 tablet_bootstrap.cc:654] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:08.661051 10222 log.cc:824] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:08.662757 10220 raft_consensus.cc:348] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.663342 10220 raft_consensus.cc:374] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:08.663482 10220 raft_consensus.cc:729] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a405b2056cff4187850eb418ce6624af, State: Initialized, Role: FOLLOWER
I20260620 12:18:08.664021 10220 consensus_queue.cc:260] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.664860 10221 tablet_bootstrap.cc:492] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: No bootstrap required, opened a new log
I20260620 12:18:08.664994 10069 heartbeater.cc:503] Master 127.24.141.62:40819 was elected leader, sending a full tablet report...
I20260620 12:18:08.665207 10221 ts_tablet_manager.cc:1397] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260620 12:18:08.665305 10220 ts_tablet_manager.cc:1428] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:18:08.669267 10221 raft_consensus.cc:348] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.669763 10221 raft_consensus.cc:374] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:08.669889 10221 raft_consensus.cc:729] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c15adf072434c9daba9e5197d7bc675, State: Initialized, Role: FOLLOWER
I20260620 12:18:08.670569 10221 consensus_queue.cc:260] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.671978 10203 heartbeater.cc:503] Master 127.24.141.62:40819 was elected leader, sending a full tablet report...
I20260620 12:18:08.672128 10221 ts_tablet_manager.cc:1428] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:18:08.673189 10222 tablet_bootstrap.cc:492] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5: No bootstrap required, opened a new log
I20260620 12:18:08.673501 10222 ts_tablet_manager.cc:1397] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260620 12:18:08.678241 10222 raft_consensus.cc:348] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.678632 10222 raft_consensus.cc:374] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:08.678709 10222 raft_consensus.cc:729] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc139f2054204a3fb739bd5cfe1e5ff5, State: Initialized, Role: FOLLOWER
I20260620 12:18:08.679311 10222 consensus_queue.cc:260] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.680294 9935 heartbeater.cc:503] Master 127.24.141.62:40819 was elected leader, sending a full tablet report...
I20260620 12:18:08.680704 10222 ts_tablet_manager.cc:1428] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260620 12:18:08.731117 10227 raft_consensus.cc:482] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:18:08.731516 10227 raft_consensus.cc:504] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.732869 10227 leader_election.cc:288] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a405b2056cff4187850eb418ce6624af (127.24.141.2:42889), cc139f2054204a3fb739bd5cfe1e5ff5 (127.24.141.1:42531)
I20260620 12:18:08.739114 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.733341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:56155 (local address 127.24.141.2:42889)
0620 12:18:08.733546 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.733551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.733700 (+ 149us) server_negotiation.cc:407] Connection header received
0620 12:18:08.733885 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.733889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.733957 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.734085 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:08.734690 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.735556 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.737554 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.738247 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.738402 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.738470 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.738831 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.738924 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 12:18:08.739108 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.733454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:52317 (local address 127.24.141.1:42531)
0620 12:18:08.733650 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.733655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.733868 (+ 213us) server_negotiation.cc:407] Connection header received
0620 12:18:08.733994 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.733997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.734051 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.734124 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:08.734690 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.735554 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.737554 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.738247 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.738402 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.738470 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.738831 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.738924 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:18:08.739326 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.733192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:42889 (local address 127.24.141.3:56155)
0620 12:18:08.733609 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.733626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.733745 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.734162 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.734167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.734204 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:08.734521 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.734534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.735771 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.735777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.737366 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.737396 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.738477 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.738483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.738673 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.738715 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.738810 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.739073 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":330}
I20260620 12:18:08.739341 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.733280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:42531 (local address 127.24.141.3:52317)
0620 12:18:08.733771 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.733785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:08.733879 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:08.734173 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:08.734175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:08.734209 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:08.734420 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.734522 (+ 102us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.735771 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.735777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.737366 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:08.737390 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.738477 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.738483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:08.738668 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.738719 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 12:18:08.738806 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:18:08.739161 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":135,"spinlock_wait_cycles":28032,"thread_start_us":188,"threads_started":1}
I20260620 12:18:08.740290 10023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3899ce0b8e488593c10d8b14b3356e" candidate_uuid: "5c15adf072434c9daba9e5197d7bc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405b2056cff4187850eb418ce6624af" is_pre_election: true
I20260620 12:18:08.740482 9889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3899ce0b8e488593c10d8b14b3356e" candidate_uuid: "5c15adf072434c9daba9e5197d7bc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" is_pre_election: true
I20260620 12:18:08.740695 10023 raft_consensus.cc:2454] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c15adf072434c9daba9e5197d7bc675 in term 0.
I20260620 12:18:08.740865 9889 raft_consensus.cc:2454] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c15adf072434c9daba9e5197d7bc675 in term 0.
I20260620 12:18:08.741415 10092 leader_election.cc:302] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c15adf072434c9daba9e5197d7bc675, cc139f2054204a3fb739bd5cfe1e5ff5; no voters:
I20260620 12:18:08.741796 10227 raft_consensus.cc:2781] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:18:08.741945 10227 raft_consensus.cc:482] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:18:08.742004 10227 raft_consensus.cc:3037] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:08.743232 10227 raft_consensus.cc:504] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.743654 10227 leader_election.cc:288] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [CANDIDATE]: Term 1 election: Requested vote from peers a405b2056cff4187850eb418ce6624af (127.24.141.2:42889), cc139f2054204a3fb739bd5cfe1e5ff5 (127.24.141.1:42531)
I20260620 12:18:08.744120 10023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3899ce0b8e488593c10d8b14b3356e" candidate_uuid: "5c15adf072434c9daba9e5197d7bc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405b2056cff4187850eb418ce6624af"
I20260620 12:18:08.744128 9889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3899ce0b8e488593c10d8b14b3356e" candidate_uuid: "5c15adf072434c9daba9e5197d7bc675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5"
I20260620 12:18:08.744274 9889 raft_consensus.cc:3037] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:08.744257 10023 raft_consensus.cc:3037] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:08.745554 10023 raft_consensus.cc:2454] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c15adf072434c9daba9e5197d7bc675 in term 1.
I20260620 12:18:08.745741 9889 raft_consensus.cc:2454] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c15adf072434c9daba9e5197d7bc675 in term 1.
I20260620 12:18:08.745975 10091 leader_election.cc:302] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c15adf072434c9daba9e5197d7bc675, a405b2056cff4187850eb418ce6624af; no voters:
I20260620 12:18:08.746191 10227 raft_consensus.cc:2781] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:08.746436 10227 raft_consensus.cc:686] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 1 LEADER]: Becoming Leader. State: Replica: 5c15adf072434c9daba9e5197d7bc675, State: Running, Role: LEADER
I20260620 12:18:08.746807 10227 consensus_queue.cc:237] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } }
I20260620 12:18:08.750689 9742 catalog_manager.cc:5495] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c15adf072434c9daba9e5197d7bc675 (127.24.141.3). New cstate: current_term: 1 leader_uuid: "5c15adf072434c9daba9e5197d7bc675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c15adf072434c9daba9e5197d7bc675" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 37257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 } health_report { overall_health: UNKNOWN } } }
I20260620 12:18:08.768198 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.763676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.24.141.3:37257)
0620 12:18:08.763830 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:08.763834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:08.763851 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:08.763953 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:08.763956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:08.764021 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:08.764151 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:08.764503 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.765172 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.766252 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:08.766513 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:08.766695 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:08.767052 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:08.767719 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:08.767830 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:08.767937 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 12:18:08.768017 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:18:08.776316 10023 raft_consensus.cc:1261] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 1 FOLLOWER]: Refusing update from remote peer 5c15adf072434c9daba9e5197d7bc675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:08.776404 9889 raft_consensus.cc:1261] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 1 FOLLOWER]: Refusing update from remote peer 5c15adf072434c9daba9e5197d7bc675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:08.777280 10227 consensus_queue.cc:1035] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc139f2054204a3fb739bd5cfe1e5ff5" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 42531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:08.777730 10230 consensus_queue.cc:1035] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a405b2056cff4187850eb418ce6624af" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:08.787660 10233 mvcc.cc:205] Tried to move back new op lower bound from 7298899512418570240 to 7298899512310075392. Current Snapshot: MvccSnapshot[applied={T|T < 7298899512418570240}]
I20260620 12:18:08.790617 10235 mvcc.cc:205] Tried to move back new op lower bound from 7298899512418570240 to 7298899512310075392. Current Snapshot: MvccSnapshot[applied={T|T < 7298899512418570240 or (T in {7298899512418570240})}]
I20260620 12:18:08.799818 9741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:08.800057 9741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:08.803421 9741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 489901dfe4fa4f33a9180c64a149d20d: Sending DeleteTablet for 3 replicas of tablet bf3899ce0b8e488593c10d8b14b3356e
I20260620 12:18:08.804523 10003 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3899ce0b8e488593c10d8b14b3356e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:08 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:49038
I20260620 12:18:08.804759 9869 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3899ce0b8e488593c10d8b14b3356e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:08 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:37744
I20260620 12:18:08.804726 10137 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3899ce0b8e488593c10d8b14b3356e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:08 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:54268
I20260620 12:18:08.805500 10243 tablet_replica.cc:330] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: stopping tablet replica
I20260620 12:18:08.805527 10244 tablet_replica.cc:330] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5: stopping tablet replica
I20260620 12:18:08.805977 10243 raft_consensus.cc:2229] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:08.806099 10244 raft_consensus.cc:2229] T bf3899ce0b8e488593c10d8b14b3356e P cc139f2054204a3fb739bd5cfe1e5ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:08.806205 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 9806
I20260620 12:18:08.807909 10243 raft_consensus.cc:2258] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:08.810041 10243 ts_tablet_manager.cc:1905] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:08.810830 10245 tablet_replica.cc:330] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: stopping tablet replica
I20260620 12:18:08.811307 10245 raft_consensus.cc:2229] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:18:08.811841 10245 raft_consensus.cc:2258] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:08.813699 10243 ts_tablet_manager.cc:1918] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:08.813903 10243 log.cc:1196] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/bf3899ce0b8e488593c10d8b14b3356e
I20260620 12:18:08.813882 10245 ts_tablet_manager.cc:1905] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:08.814389 10243 ts_tablet_manager.cc:1939] T bf3899ce0b8e488593c10d8b14b3356e P a405b2056cff4187850eb418ce6624af: Deleting consensus metadata
I20260620 12:18:08.816310 9729 catalog_manager.cc:4841] TS a405b2056cff4187850eb418ce6624af (127.24.141.2:42889): tablet bf3899ce0b8e488593c10d8b14b3356e (table test-table [id=8490e738aa0e48c8ba8f88701fdafbdc]) successfully deleted
W20260620 12:18:08.816437 9730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:42531 (error 108)
I20260620 12:18:08.817281 10245 ts_tablet_manager.cc:1918] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:08.817430 10245 log.cc:1196] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/bf3899ce0b8e488593c10d8b14b3356e
I20260620 12:18:08.817435 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 9940
I20260620 12:18:08.819833 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:08.819429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:42531 (local address 127.0.0.1:37748)
0620 12:18:08.819638 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:08.819714 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 12:18:08.820173 10245 ts_tablet_manager.cc:1939] T bf3899ce0b8e488593c10d8b14b3356e P 5c15adf072434c9daba9e5197d7bc675: Deleting consensus metadata
W20260620 12:18:08.820119 9730 catalog_manager.cc:4584] TS cc139f2054204a3fb739bd5cfe1e5ff5 (127.24.141.1:42531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf3899ce0b8e488593c10d8b14b3356e: Network error: Client connection negotiation failed: client connection to 127.24.141.1:42531: connect: Connection refused (error 111)
I20260620 12:18:08.821911 9730 catalog_manager.cc:4841] TS 5c15adf072434c9daba9e5197d7bc675 (127.24.141.3:37257): tablet bf3899ce0b8e488593c10d8b14b3356e (table test-table [id=8490e738aa0e48c8ba8f88701fdafbdc]) successfully deleted
I20260620 12:18:08.825732 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 10074
I20260620 12:18:08.832975 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 9712
2026-06-20T12:18:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2492 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T12:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:08Z Disabled control of system clock
I20260620 12:18:08.869156 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43225
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:43225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:18:09.004220 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:09.004707 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:09.004779 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:09.009477 10250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:09.009583 10250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:09.009599 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:09.009617 10250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:09.009632 10250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:09.015837 10250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:43225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:09.017593 10250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:09.019470 10250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:09.030267 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:09.030279 10258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:09.030279 10256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:09.030550 10250 server_base.cc:1034] running on GCE node
I20260620 12:18:09.031528 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:09.033023 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:09.034313 10250 hybrid_clock.cc:648] HybridClock initialized: now 1781957889034263 us; error 87 us; skew 500 ppm
I20260620 12:18:09.036821 10250 webserver.cc:460] Webserver started at http://127.24.141.62:46121/ using document root <none> and password file <none>
I20260620 12:18:09.037640 10250 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:09.037725 10250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:09.037962 10250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:09.040997 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "b3f3cc5fca174b06aa15b6e3685c66ff"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "a11b3363a30821a1eb94c7745500bed7"
server_key_iv: "7dd63333cf01b3447fb705597d597943"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.041819 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "b3f3cc5fca174b06aa15b6e3685c66ff"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "a11b3363a30821a1eb94c7745500bed7"
server_key_iv: "7dd63333cf01b3447fb705597d597943"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.046804 10250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:18:09.050239 10264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.052069 10250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:18:09.052275 10250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "b3f3cc5fca174b06aa15b6e3685c66ff"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "a11b3363a30821a1eb94c7745500bed7"
server_key_iv: "7dd63333cf01b3447fb705597d597943"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.052419 10250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:09.076642 10250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:09.077543 10250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:09.077766 10250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:09.086083 10250 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:43225
I20260620 12:18:09.086108 10316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:43225 every 8 connection(s)
I20260620 12:18:09.087249 10250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:09.088385 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 10250
I20260620 12:18:09.088585 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:09.088966 25140 external_mini_cluster.cc:1428] Setting key 8b31194989220b8bc1beed5e7f2a94fd
I20260620 12:18:09.092231 10317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:09.099895 10317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff: Bootstrap starting.
I20260620 12:18:09.101634 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.090970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.24.141.62:43225)
0620 12:18:09.092017 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.092037 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.092091 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:18:09.093043 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.093103 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.093716 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.094120 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:09.094646 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.095736 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.096891 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.097844 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.098052 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.099053 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:09.099059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:09.099078 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:09.099718 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:09.099794 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:09.099820 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:09.099947 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.100111 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.100497 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":210,"threads_started":1}
I20260620 12:18:09.103281 10317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:09.104190 10317 log.cc:824] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:09.106824 10317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff: No bootstrap required, opened a new log
I20260620 12:18:09.111114 10317 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43225 } }
I20260620 12:18:09.111400 10317 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:09.111457 10317 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f3cc5fca174b06aa15b6e3685c66ff, State: Initialized, Role: FOLLOWER
I20260620 12:18:09.111945 10317 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43225 } }
I20260620 12:18:09.112123 10317 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:09.112195 10317 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:09.112308 10317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:09.113559 10317 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43225 } }
I20260620 12:18:09.113960 10317 leader_election.cc:302] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f3cc5fca174b06aa15b6e3685c66ff; no voters:
I20260620 12:18:09.114357 10317 leader_election.cc:288] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:09.114579 10322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:09.114900 10322 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [term 1 LEADER]: Becoming Leader. State: Replica: b3f3cc5fca174b06aa15b6e3685c66ff, State: Running, Role: LEADER
I20260620 12:18:09.115650 10322 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43225 } }
I20260620 12:18:09.115942 10317 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:09.117630 10324 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f3cc5fca174b06aa15b6e3685c66ff. Latest consensus state: current_term: 1 leader_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43225 } } }
I20260620 12:18:09.117944 10324 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:09.118317 10323 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f3cc5fca174b06aa15b6e3685c66ff" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43225 } } }
I20260620 12:18:09.118462 10323 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff [sys.catalog]: This master's current role is: LEADER
W20260620 12:18:09.122148 10337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:09.122267 10337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:09.122375 10336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:09.123298 10336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:09.130795 10336 catalog_manager.cc:1347] Generated new cluster ID: 06a68ea101d3454d93cf4b2608e8c3c6
I20260620 12:18:09.130901 10336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:09.162636 10336 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:09.163794 10336 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:09.174391 10336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff: Generated new TSK 0
I20260620 12:18:09.175551 10336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:09.189906 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43225
--builtin_ntp_servers=127.24.141.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:18:09.326874 10341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:09.327296 10341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:09.327344 10341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:09.331800 10341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:09.331892 10341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:09.331956 10341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:18:09.337365 10341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:09.338891 10341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:09.340662 10341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:09.352468 10346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:09.352480 10349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:09.352468 10347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:09.352854 10341 server_base.cc:1034] running on GCE node
I20260620 12:18:09.353482 10341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:09.354293 10341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:09.355523 10341 hybrid_clock.cc:648] HybridClock initialized: now 1781957889355478 us; error 56 us; skew 500 ppm
I20260620 12:18:09.358003 10341 webserver.cc:460] Webserver started at http://127.24.141.1:43675/ using document root <none> and password file <none>
I20260620 12:18:09.358732 10341 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:09.358832 10341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:09.359056 10341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:09.361611 10341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "affa7b1ac1a34313b9f81328328aa02d"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "448ff93ea045c40caff829b62c94a9b0"
server_key_iv: "bfe2dde0983ec4c776aa0ab67584f75c"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.362294 10341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "affa7b1ac1a34313b9f81328328aa02d"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "448ff93ea045c40caff829b62c94a9b0"
server_key_iv: "bfe2dde0983ec4c776aa0ab67584f75c"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.367106 10341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:09.370420 10355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.372248 10341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:18:09.372452 10341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "affa7b1ac1a34313b9f81328328aa02d"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "448ff93ea045c40caff829b62c94a9b0"
server_key_iv: "bfe2dde0983ec4c776aa0ab67584f75c"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.372597 10341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:09.390959 10341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:09.391773 10341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:09.391976 10341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:09.392645 10341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:09.393893 10341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:09.393949 10341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.394009 10341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:09.394038 10341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.404469 10341 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:43343
I20260620 12:18:09.404539 10468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:43343 every 8 connection(s)
I20260620 12:18:09.405875 10341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:18:09.408869 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 10341
I20260620 12:18:09.409011 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:18:09.409526 25140 external_mini_cluster.cc:1428] Setting key 6ea5d3148a6fee2685d2039c06be839a
I20260620 12:18:09.412369 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43225
--builtin_ntp_servers=127.24.141.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:18:09.418512 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.408625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:55499 (local address 127.24.141.62:43225)
0620 12:18:09.408813 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.408817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.410852 (+ 2035us) server_negotiation.cc:407] Connection header received
0620 12:18:09.412727 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.412731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.412793 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.412876 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:09.414185 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.414786 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.416160 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.416391 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.416494 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.417858 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:09.417861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:09.417864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:09.417880 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:09.417903 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:09.417912 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:09.417974 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.418153 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.418287 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:18:09.419589 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.409480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43225 (local address 127.24.141.1:55499)
0620 12:18:09.410640 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:09.410719 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:09.412425 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:09.413177 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:09.413186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:09.413309 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:09.414009 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.414026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.414926 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.414930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.416026 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.416039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.416567 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.416571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.416785 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.417535 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:09.417537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:09.417593 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:09.417692 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:09.417991 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:09.418001 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:09.418009 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:18:09.418133 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 12:18:09.418513 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":225,"threads_started":1}
I20260620 12:18:09.421259 10469 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43225
I20260620 12:18:09.421555 10469 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:09.422221 10469 heartbeater.cc:511] Master 127.24.141.62:43225 requested a full tablet report, sending...
I20260620 12:18:09.424294 10281 ts_manager.cc:194] Registered new tserver with Master: affa7b1ac1a34313b9f81328328aa02d (127.24.141.1:43343)
I20260620 12:18:09.425768 10281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.1:55499
W20260620 12:18:09.548993 10472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:09.549467 10472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:09.549531 10472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:09.553931 10472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:09.554018 10472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:09.554098 10472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:18:09.559508 10472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:09.560779 10472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:09.562459 10472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:09.573860 10480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:09.574132 10472 server_base.cc:1034] running on GCE node
W20260620 12:18:09.573873 10477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:09.573873 10478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:09.575181 10472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:09.576097 10472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:09.577304 10472 hybrid_clock.cc:648] HybridClock initialized: now 1781957889577271 us; error 58 us; skew 500 ppm
I20260620 12:18:09.579841 10472 webserver.cc:460] Webserver started at http://127.24.141.2:41125/ using document root <none> and password file <none>
I20260620 12:18:09.580610 10472 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:09.580706 10472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:09.580964 10472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:09.583648 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "88f1ea51e57842d4add438b065f6643d"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "6577985435e7b32e790b9d61eb10dfde"
server_key_iv: "dd5266c0935490b6f7cf7e06a5cc66a4"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.584319 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "88f1ea51e57842d4add438b065f6643d"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "6577985435e7b32e790b9d61eb10dfde"
server_key_iv: "dd5266c0935490b6f7cf7e06a5cc66a4"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.589320 10472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:18:09.593130 10486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.595232 10472 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:09.595445 10472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "88f1ea51e57842d4add438b065f6643d"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "6577985435e7b32e790b9d61eb10dfde"
server_key_iv: "dd5266c0935490b6f7cf7e06a5cc66a4"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.595623 10472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:09.615798 10472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:09.616869 10472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:09.617156 10472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:09.618028 10472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:09.619550 10472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:09.619623 10472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.619695 10472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:09.619725 10472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.631749 10472 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:37025
I20260620 12:18:09.631862 10599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:37025 every 8 connection(s)
I20260620 12:18:09.633031 10472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:18:09.641274 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 10472
I20260620 12:18:09.641410 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:18:09.641713 25140 external_mini_cluster.cc:1428] Setting key 4f5db27e1fcd99045321b74bc13af5f4
I20260620 12:18:09.644760 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43225
--builtin_ntp_servers=127.24.141.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:18:09.646617 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.635445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:55609 (local address 127.24.141.62:43225)
0620 12:18:09.635634 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.635638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.637215 (+ 1577us) server_negotiation.cc:407] Connection header received
0620 12:18:09.639106 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.639217 (+ 111us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.639340 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.639475 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:09.641024 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.641703 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.643688 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.644121 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.644273 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.646025 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:09.646028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:09.646031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:09.646049 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:09.646077 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:09.646087 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:09.646155 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.646345 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.646441 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:18:09.647876 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.635861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43225 (local address 127.24.141.2:55609)
0620 12:18:09.636919 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:09.636989 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:09.638734 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:09.639802 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:09.639813 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:09.639953 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:09.640729 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.640747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.641928 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.641933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.643460 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.643480 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.644392 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.644399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.644633 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.645553 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:09.645556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:09.645623 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:09.645759 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:09.646176 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:09.646194 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:09.646206 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 12:18:09.646337 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 12:18:09.646757 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":168,"threads_started":1}
I20260620 12:18:09.649893 10600 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43225
I20260620 12:18:09.650333 10600 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:09.651350 10600 heartbeater.cc:511] Master 127.24.141.62:43225 requested a full tablet report, sending...
I20260620 12:18:09.653374 10281 ts_manager.cc:194] Registered new tserver with Master: 88f1ea51e57842d4add438b065f6643d (127.24.141.2:37025)
I20260620 12:18:09.654100 10281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.2:55609
W20260620 12:18:09.784979 10603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:09.785384 10603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:09.785429 10603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:09.790033 10603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:09.790124 10603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:09.790189 10603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:18:09.795506 10603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:09.797020 10603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:09.798812 10603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:09.810846 10608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:09.810954 10609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:09.810846 10611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:09.811172 10603 server_base.cc:1034] running on GCE node
I20260620 12:18:09.811861 10603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:09.812688 10603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:09.813875 10603 hybrid_clock.cc:648] HybridClock initialized: now 1781957889813845 us; error 66 us; skew 500 ppm
I20260620 12:18:09.816323 10603 webserver.cc:460] Webserver started at http://127.24.141.3:42499/ using document root <none> and password file <none>
I20260620 12:18:09.817204 10603 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:09.817301 10603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:09.817602 10603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:09.820576 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "f05e0d52f130538084df2a34f3746876"
server_key_iv: "9912a8e36c79eeee805562dcc3e50e50"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.821326 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "f05e0d52f130538084df2a34f3746876"
server_key_iv: "9912a8e36c79eeee805562dcc3e50e50"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.826577 10603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:09.830387 10617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.832485 10603 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:18:09.832696 10603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f"
format_stamp: "Formatted at 2026-06-20 12:18:09 on dist-test-slave-rh5v"
server_key: "f05e0d52f130538084df2a34f3746876"
server_key_iv: "9912a8e36c79eeee805562dcc3e50e50"
server_key_version: "encryptionkey@0"
I20260620 12:18:09.832888 10603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:09.847208 10603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:09.848106 10603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:09.848306 10603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:09.849046 10603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:09.850306 10603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:09.850369 10603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.850433 10603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:09.850461 10603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:09.861590 10603 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:36405
I20260620 12:18:09.861698 10730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:36405 every 8 connection(s)
I20260620 12:18:09.862880 10603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:18:09.864207 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 10603
I20260620 12:18:09.864327 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:18:09.864622 25140 external_mini_cluster.cc:1428] Setting key da742778db1a79aaaef5001ed95e425c
I20260620 12:18:09.875303 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.865597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:56341 (local address 127.24.141.62:43225)
0620 12:18:09.865775 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.865780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.867502 (+ 1722us) server_negotiation.cc:407] Connection header received
0620 12:18:09.869233 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.869237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.869311 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.869404 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:09.870679 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.871276 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.872737 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.873011 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.873227 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.874758 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:09.874761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:09.874765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:09.874781 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:09.874805 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:09.874813 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:09.874877 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.875044 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.875127 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:18:09.876379 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.866208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:43225 (local address 127.24.141.3:56341)
0620 12:18:09.867267 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:09.867338 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:09.868868 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:09.869633 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:09.869644 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:09.869789 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:09.870469 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.870486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.871438 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.871440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.872543 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.872555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.873343 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.873351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.873635 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.874373 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:09.874375 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:09.874440 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:09.874542 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:09.874897 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:09.874907 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:09.874913 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:18:09.875039 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 12:18:09.875458 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":227,"threads_started":1}
I20260620 12:18:09.878218 10731 heartbeater.cc:347] Connected to a master server at 127.24.141.62:43225
I20260620 12:18:09.878621 10731 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:09.879446 10731 heartbeater.cc:511] Master 127.24.141.62:43225 requested a full tablet report, sending...
I20260620 12:18:09.880734 10281 ts_manager.cc:194] Registered new tserver with Master: d8ef0ee5ec5b4470bd433e0cf22f844f (127.24.141.3:36405)
I20260620 12:18:09.881455 10281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.3:56341
I20260620 12:18:09.892228 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:18:09.898973 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.894868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.24.141.62:43225)
0620 12:18:09.895088 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.895093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.895298 (+ 205us) server_negotiation.cc:407] Connection header received
0620 12:18:09.895453 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.895457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.895510 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.895582 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:09.895918 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.896450 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.897387 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.897606 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.897709 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.898552 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:09.898555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:09.898557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:09.898572 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:09.898591 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:09.898599 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:09.898658 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.898749 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.898812 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 12:18:09.903311 10281 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:18:09.907394 10281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:18:09.926435 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.918973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:37025 (local address 127.0.0.1:52174)
0620 12:18:09.920527 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:09.920542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:09.920632 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:09.922091 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:09.922094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:09.922112 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:09.922225 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.922231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.923512 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.923515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.924688 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.924699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.925725 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.925729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.925886 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.925939 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 12:18:09.926082 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 12:18:09.926249 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":123,"thread_start_us":110,"threads_started":1}
I20260620 12:18:09.926997 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.918964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36405 (local address 127.0.0.1:52964)
0620 12:18:09.920033 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:09.920103 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:09.920268 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:09.921376 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:09.921383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:09.921420 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:09.921728 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.921737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.923332 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.923336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.925145 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.925172 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.926572 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.926576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.926677 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.926702 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:18:09.926785 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 12:18:09.926866 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":76,"spinlock_wait_cycles":6528,"thread_start_us":114,"threads_started":1}
I20260620 12:18:09.927381 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.920167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.24.141.2:37025)
0620 12:18:09.921674 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.921686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.921709 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:18:09.921793 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.921798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.921954 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.922095 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:09.922337 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.923376 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.924825 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.925553 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.925674 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.925912 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.927013 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.927196 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":106,"threads_started":1}
I20260620 12:18:09.927419 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.920040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.24.141.3:36405)
0620 12:18:09.920809 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.920823 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.920849 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:18:09.920929 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.920934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.921178 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.921337 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:09.921914 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.923173 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.925361 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.926179 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.926282 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.926504 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.927037 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.927190 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":176,"threads_started":1}
I20260620 12:18:09.927860 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.919493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.24.141.1:43343)
0620 12:18:09.920580 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:09.920589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:09.920663 (+ 74us) server_negotiation.cc:407] Connection header received
0620 12:18:09.920914 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:09.920921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:09.921235 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:09.921456 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:09.921871 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.923153 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.925359 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.926173 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.926297 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.926502 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:09.927533 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0620 12:18:09.927701 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":87,"threads_started":1}
I20260620 12:18:09.928872 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:09.919311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43343 (local address 127.0.0.1:52986)
0620 12:18:09.920551 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:09.920569 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:09.920705 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:09.921423 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:09.921426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:09.921445 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:09.921718 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.921727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.923332 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.923336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.925145 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:09.925167 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:09.926955 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:09.926958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:09.927057 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:09.927085 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:18:09.927417 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 12:18:09.928748 (+ 1331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"mutex_wait_us":371,"thread_start_us":145,"threads_started":1}
I20260620 12:18:09.932175 10403 tablet_service.cc:1467] Processing CreateTablet for tablet a1efdbe91aff4d2ea481844c9e4b7a03 (DEFAULT_TABLE table=test-table [id=b07954d226d84f08997d6610c340eace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:09.931840 10665 tablet_service.cc:1467] Processing CreateTablet for tablet a1efdbe91aff4d2ea481844c9e4b7a03 (DEFAULT_TABLE table=test-table [id=b07954d226d84f08997d6610c340eace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:09.931840 10534 tablet_service.cc:1467] Processing CreateTablet for tablet a1efdbe91aff4d2ea481844c9e4b7a03 (DEFAULT_TABLE table=test-table [id=b07954d226d84f08997d6610c340eace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:09.933468 10403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1efdbe91aff4d2ea481844c9e4b7a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:09.933573 10534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1efdbe91aff4d2ea481844c9e4b7a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:09.933573 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1efdbe91aff4d2ea481844c9e4b7a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:09.943054 10749 tablet_bootstrap.cc:492] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: Bootstrap starting.
I20260620 12:18:09.945456 10748 tablet_bootstrap.cc:492] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d: Bootstrap starting.
I20260620 12:18:09.945631 10750 tablet_bootstrap.cc:492] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Bootstrap starting.
I20260620 12:18:09.945715 10749 tablet_bootstrap.cc:654] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:09.946784 10749 log.cc:824] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:09.947726 10748 tablet_bootstrap.cc:654] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:09.948148 10750 tablet_bootstrap.cc:654] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:09.948596 10748 log.cc:824] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:09.949252 10750 log.cc:824] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:09.949973 10749 tablet_bootstrap.cc:492] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: No bootstrap required, opened a new log
I20260620 12:18:09.950219 10749 ts_tablet_manager.cc:1397] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:18:09.951050 10748 tablet_bootstrap.cc:492] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d: No bootstrap required, opened a new log
I20260620 12:18:09.951282 10748 ts_tablet_manager.cc:1397] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:18:09.951805 10750 tablet_bootstrap.cc:492] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: No bootstrap required, opened a new log
I20260620 12:18:09.952047 10750 ts_tablet_manager.cc:1397] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 12:18:09.955202 10748 raft_consensus.cc:348] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:09.955514 10748 raft_consensus.cc:374] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:09.955592 10748 raft_consensus.cc:729] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa7b1ac1a34313b9f81328328aa02d, State: Initialized, Role: FOLLOWER
I20260620 12:18:09.955513 10749 raft_consensus.cc:348] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:09.955978 10749 raft_consensus.cc:374] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:09.956056 10749 raft_consensus.cc:729] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f1ea51e57842d4add438b065f6643d, State: Initialized, Role: FOLLOWER
I20260620 12:18:09.956192 10748 consensus_queue.cc:260] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:09.956647 10749 consensus_queue.cc:260] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:09.957175 10469 heartbeater.cc:503] Master 127.24.141.62:43225 was elected leader, sending a full tablet report...
I20260620 12:18:09.957041 10750 raft_consensus.cc:348] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:09.957458 10750 raft_consensus.cc:374] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:09.957511 10750 raft_consensus.cc:729] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ef0ee5ec5b4470bd433e0cf22f844f, State: Initialized, Role: FOLLOWER
I20260620 12:18:09.958031 10750 consensus_queue.cc:260] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:09.958125 10749 ts_tablet_manager.cc:1428] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20260620 12:18:09.958603 10600 heartbeater.cc:503] Master 127.24.141.62:43225 was elected leader, sending a full tablet report...
I20260620 12:18:09.958989 10748 ts_tablet_manager.cc:1428] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260620 12:18:09.959371 10750 ts_tablet_manager.cc:1428] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260620 12:18:09.959803 10731 heartbeater.cc:503] Master 127.24.141.62:43225 was elected leader, sending a full tablet report...
W20260620 12:18:10.114177 10732 tablet.cc:2369] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:10.118443 10755 raft_consensus.cc:482] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:18:10.118781 10755 raft_consensus.cc:504] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:10.120339 10755 leader_election.cc:288] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ef0ee5ec5b4470bd433e0cf22f844f (127.24.141.3:36405), affa7b1ac1a34313b9f81328328aa02d (127.24.141.1:43343)
I20260620 12:18:10.126423 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:10.120671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36405 (local address 127.24.141.2:49891)
0620 12:18:10.120921 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:10.120938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:10.121093 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:10.121484 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:10.121486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:10.121501 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:10.121649 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:10.121661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.122911 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.122916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:10.124495 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:10.124509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.125693 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.125697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:10.125927 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:10.125948 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:18:10.126055 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 12:18:10.126185 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":193}
I20260620 12:18:10.126430 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:10.120840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43343 (local address 127.24.141.2:51731)
0620 12:18:10.121266 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:10.121277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:10.121368 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:10.121593 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:10.121597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:10.121611 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:10.121719 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:10.121726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.122911 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.122916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:10.124495 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:10.124509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.125693 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.125697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:10.125903 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:10.125947 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 12:18:10.126055 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:18:10.126185 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":127,"thread_start_us":157,"threads_started":1}
I20260620 12:18:10.126544 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:10.120828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:49891 (local address 127.24.141.3:36405)
0620 12:18:10.120965 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:10.120970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:10.120989 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:18:10.121273 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:10.121276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:10.121346 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:10.121455 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:10.121880 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.122741 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.124740 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.125532 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.125679 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:10.125748 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:10.126217 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 12:18:10.126373 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:18:10.126544 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:10.120943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:51731 (local address 127.24.141.1:43343)
0620 12:18:10.121122 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:10.121127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:10.121371 (+ 244us) server_negotiation.cc:407] Connection header received
0620 12:18:10.121441 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:10.121445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:10.121491 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:10.121571 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:10.121880 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.122741 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.124741 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.125498 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.125679 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:10.125748 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:10.126225 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 12:18:10.126373 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:18:10.127652 10685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1efdbe91aff4d2ea481844c9e4b7a03" candidate_uuid: "88f1ea51e57842d4add438b065f6643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" is_pre_election: true
I20260620 12:18:10.128052 10685 raft_consensus.cc:2454] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88f1ea51e57842d4add438b065f6643d in term 0.
I20260620 12:18:10.128057 10423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1efdbe91aff4d2ea481844c9e4b7a03" candidate_uuid: "88f1ea51e57842d4add438b065f6643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa7b1ac1a34313b9f81328328aa02d" is_pre_election: true
I20260620 12:18:10.128458 10423 raft_consensus.cc:2454] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88f1ea51e57842d4add438b065f6643d in term 0.
I20260620 12:18:10.128793 10488 leader_election.cc:302] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88f1ea51e57842d4add438b065f6643d, d8ef0ee5ec5b4470bd433e0cf22f844f; no voters:
I20260620 12:18:10.129194 10755 raft_consensus.cc:2781] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:18:10.129298 10755 raft_consensus.cc:482] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:18:10.129341 10755 raft_consensus.cc:3037] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:10.130609 10755 raft_consensus.cc:504] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:10.131033 10755 leader_election.cc:288] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [CANDIDATE]: Term 1 election: Requested vote from peers d8ef0ee5ec5b4470bd433e0cf22f844f (127.24.141.3:36405), affa7b1ac1a34313b9f81328328aa02d (127.24.141.1:43343)
I20260620 12:18:10.131577 10685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1efdbe91aff4d2ea481844c9e4b7a03" candidate_uuid: "88f1ea51e57842d4add438b065f6643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f"
I20260620 12:18:10.131680 10423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1efdbe91aff4d2ea481844c9e4b7a03" candidate_uuid: "88f1ea51e57842d4add438b065f6643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa7b1ac1a34313b9f81328328aa02d"
I20260620 12:18:10.131768 10685 raft_consensus.cc:3037] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:10.131839 10423 raft_consensus.cc:3037] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:10.133140 10423 raft_consensus.cc:2454] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f1ea51e57842d4add438b065f6643d in term 1.
I20260620 12:18:10.133140 10685 raft_consensus.cc:2454] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f1ea51e57842d4add438b065f6643d in term 1.
I20260620 12:18:10.133636 10490 leader_election.cc:302] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88f1ea51e57842d4add438b065f6643d, affa7b1ac1a34313b9f81328328aa02d; no voters:
I20260620 12:18:10.133908 10755 raft_consensus.cc:2781] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:10.134169 10755 raft_consensus.cc:686] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 1 LEADER]: Becoming Leader. State: Replica: 88f1ea51e57842d4add438b065f6643d, State: Running, Role: LEADER
W20260620 12:18:10.134482 10601 tablet.cc:2369] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:10.134687 10755 consensus_queue.cc:237] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } }
I20260620 12:18:10.139273 10277 catalog_manager.cc:5495] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d reported cstate change: term changed from 0 to 1, leader changed from <none> to 88f1ea51e57842d4add438b065f6643d (127.24.141.2). New cstate: current_term: 1 leader_uuid: "88f1ea51e57842d4add438b065f6643d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88f1ea51e57842d4add438b065f6643d" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 37025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 } health_report { overall_health: UNKNOWN } } }
W20260620 12:18:10.157635 10470 tablet.cc:2369] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:10.171725 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:10.166445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.24.141.2:37025)
0620 12:18:10.166663 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:10.166667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:10.166690 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:18:10.166817 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:10.166820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:10.166903 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:10.167041 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:10.167374 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.167961 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.169548 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:10.169829 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:10.169940 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:10.170432 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:10.171097 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:10.171222 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:10.171396 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 12:18:10.171479 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 12:18:10.180294 10423 raft_consensus.cc:1261] T a1efdbe91aff4d2ea481844c9e4b7a03 P affa7b1ac1a34313b9f81328328aa02d [term 1 FOLLOWER]: Refusing update from remote peer 88f1ea51e57842d4add438b065f6643d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:10.180326 10685 raft_consensus.cc:1261] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 1 FOLLOWER]: Refusing update from remote peer 88f1ea51e57842d4add438b065f6643d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:10.181219 10755 consensus_queue.cc:1035] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa7b1ac1a34313b9f81328328aa02d" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:10.181530 10758 consensus_queue.cc:1035] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ef0ee5ec5b4470bd433e0cf22f844f" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:10.204780 10280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:10.204996 10280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:10.208763 10280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3f3cc5fca174b06aa15b6e3685c66ff: Sending DeleteTablet for 3 replicas of tablet a1efdbe91aff4d2ea481844c9e4b7a03
I20260620 12:18:10.210085 10403 tablet_service.cc:1514] Processing DeleteTablet for tablet a1efdbe91aff4d2ea481844c9e4b7a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:10 UTC) from {username='slave'} at 127.0.0.1:52986
I20260620 12:18:10.210064 10665 tablet_service.cc:1514] Processing DeleteTablet for tablet a1efdbe91aff4d2ea481844c9e4b7a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:10 UTC) from {username='slave'} at 127.0.0.1:52964
I20260620 12:18:10.210294 10534 tablet_service.cc:1514] Processing DeleteTablet for tablet a1efdbe91aff4d2ea481844c9e4b7a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:10 UTC) from {username='slave'} at 127.0.0.1:52174
I20260620 12:18:10.210922 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 10341
I20260620 12:18:10.211018 10771 tablet_replica.cc:330] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: stopping tablet replica
I20260620 12:18:10.211707 10771 raft_consensus.cc:2229] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:10.212141 10771 raft_consensus.cc:2258] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:10.215544 10771 ts_tablet_manager.cc:1905] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:10.220615 10771 ts_tablet_manager.cc:1918] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:10.220811 10771 log.cc:1196] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/a1efdbe91aff4d2ea481844c9e4b7a03
I20260620 12:18:10.221197 10771 ts_tablet_manager.cc:1939] T a1efdbe91aff4d2ea481844c9e4b7a03 P d8ef0ee5ec5b4470bd433e0cf22f844f: Deleting consensus metadata
I20260620 12:18:10.221606 10773 tablet_replica.cc:330] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: stopping tablet replica
I20260620 12:18:10.222223 10773 raft_consensus.cc:2229] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:18:10.222901 10773 raft_consensus.cc:2258] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:10.223428 10266 catalog_manager.cc:4841] TS d8ef0ee5ec5b4470bd433e0cf22f844f (127.24.141.3:36405): tablet a1efdbe91aff4d2ea481844c9e4b7a03 (table test-table [id=b07954d226d84f08997d6610c340eace]) successfully deleted
W20260620 12:18:10.224922 10268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:43343 (error 108)
I20260620 12:18:10.225642 10773 ts_tablet_manager.cc:1905] T a1efdbe91aff4d2ea481844c9e4b7a03 P 88f1ea51e57842d4add438b065f6643d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:10.226845 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:10.226432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43343 (local address 127.0.0.1:52992)
0620 12:18:10.226617 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:10.226717 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 12:18:10.226931 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 10472
W20260620 12:18:10.229215 10268 catalog_manager.cc:4584] TS affa7b1ac1a34313b9f81328328aa02d (127.24.141.1:43343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1efdbe91aff4d2ea481844c9e4b7a03: Network error: Client connection negotiation failed: client connection to 127.24.141.1:43343: connect: Connection refused (error 111)
I20260620 12:18:10.234910 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 10603
I20260620 12:18:10.236343 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:10.235191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:37025 (local address 127.0.0.1:52178)
0620 12:18:10.236116 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:10.236208 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.2:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":792}
I20260620 12:18:10.241747 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 10250
2026-06-20T12:18:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1410 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:10 dist-test-slave-rh5v krb5kdc[10777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:10 dist-test-slave-rh5v krb5kdc[10777](info): set up 2 sockets
Jun 20 12:18:10 dist-test-slave-rh5v krb5kdc[10777](info): commencing operation
krb5kdc: starting...
W20260620 12:18:11.237591 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:11 dist-test-slave-rh5v krb5kdc[10777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:11.380684 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40703
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:40703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:11.519773 10790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:11.520123 10790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:11.520167 10790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:11.524470 10790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:11.524576 10790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:11.524595 10790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:11.524612 10790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:11.524634 10790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:11.531222 10790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:40703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:11.532821 10790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:11.534282 10790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:11.544942 10796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:11.544950 10798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:11.544997 10795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:11.545533 10790 server_base.cc:1034] running on GCE node
I20260620 12:18:11.546253 10790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:11.547633 10790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:11.548858 10790 hybrid_clock.cc:648] HybridClock initialized: now 1781957891548822 us; error 55 us; skew 500 ppm
Jun 20 12:18:11 dist-test-slave-rh5v krb5kdc[10777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957891, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:11.552587 10790 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:11.553902 10790 webserver.cc:460] Webserver started at http://127.24.141.62:36301/ using document root <none> and password file <none>
I20260620 12:18:11.554617 10790 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:11.554698 10790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:11.554937 10790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:11.557413 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "e59636190217440881bc9562d613d2c9"
format_stamp: "Formatted at 2026-06-20 12:18:11 on dist-test-slave-rh5v"
server_key: "b685e05bdd3018c851adc065aa304b77"
server_key_iv: "b4026e675f2e5e9c768f8676272c0dde"
server_key_version: "encryptionkey@0"
I20260620 12:18:11.558080 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "e59636190217440881bc9562d613d2c9"
format_stamp: "Formatted at 2026-06-20 12:18:11 on dist-test-slave-rh5v"
server_key: "b685e05bdd3018c851adc065aa304b77"
server_key_iv: "b4026e675f2e5e9c768f8676272c0dde"
server_key_version: "encryptionkey@0"
I20260620 12:18:11.563124 10790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:11.566588 10805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:11.568593 10790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:11.568817 10790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "e59636190217440881bc9562d613d2c9"
format_stamp: "Formatted at 2026-06-20 12:18:11 on dist-test-slave-rh5v"
server_key: "b685e05bdd3018c851adc065aa304b77"
server_key_iv: "b4026e675f2e5e9c768f8676272c0dde"
server_key_version: "encryptionkey@0"
I20260620 12:18:11.568964 10790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:11.594329 10790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:11.598196 10790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:11.598429 10790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:11.606900 10790 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:40703
I20260620 12:18:11.607013 10857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:40703 every 8 connection(s)
I20260620 12:18:11.608100 10790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:11.610998 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 10790
I20260620 12:18:11.611186 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:11.611563 25140 external_mini_cluster.cc:1428] Setting key 9cafca71f71a32e27b87ea4f801a615d
I20260620 12:18:11.612483 10858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:11.621526 10858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9: Bootstrap starting.
Jun 20 12:18:11 dist-test-slave-rh5v krb5kdc[10777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:11.624923 10858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:11.625870 10858 log.cc:824] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:11.628854 10858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9: No bootstrap required, opened a new log
I20260620 12:18:11.629842 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:11.613442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.24.141.62:40703)
0620 12:18:11.614126 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:11.614147 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:11.614200 (+ 53us) server_negotiation.cc:407] Connection header received
0620 12:18:11.615012 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:11.615036 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:11.615508 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:11.615920 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:11.616893 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:11.618091 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:11.619034 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:11.619633 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:11.619803 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:11.623412 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:11.623420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:11.623440 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:11.623485 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:11.625549 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:11.626170 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:11.626180 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:11.626187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:11.626423 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:11.626821 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:11.626825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:11.626827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:11.627478 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:11.627691 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:11.628154 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 12:18:11.628576 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":185,"threads_started":1}
I20260620 12:18:11.633843 10858 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59636190217440881bc9562d613d2c9" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40703 } }
I20260620 12:18:11.634147 10858 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:11.634193 10858 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59636190217440881bc9562d613d2c9, State: Initialized, Role: FOLLOWER
I20260620 12:18:11.634675 10858 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59636190217440881bc9562d613d2c9" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40703 } }
I20260620 12:18:11.634840 10858 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:11.634902 10858 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:11.634997 10858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:11.636122 10858 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59636190217440881bc9562d613d2c9" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40703 } }
I20260620 12:18:11.636502 10858 leader_election.cc:302] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59636190217440881bc9562d613d2c9; no voters:
I20260620 12:18:11.636854 10858 leader_election.cc:288] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:11.637189 10863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:11.637529 10863 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [term 1 LEADER]: Becoming Leader. State: Replica: e59636190217440881bc9562d613d2c9, State: Running, Role: LEADER
I20260620 12:18:11.637962 10863 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59636190217440881bc9562d613d2c9" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40703 } }
I20260620 12:18:11.638367 10858 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:11.640565 10864 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59636190217440881bc9562d613d2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59636190217440881bc9562d613d2c9" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40703 } } }
I20260620 12:18:11.640729 10864 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:11.641201 10878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:11.641212 10865 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59636190217440881bc9562d613d2c9. Latest consensus state: current_term: 1 leader_uuid: "e59636190217440881bc9562d613d2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59636190217440881bc9562d613d2c9" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40703 } } }
I20260620 12:18:11.642168 10865 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:11.645196 10878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:11.653859 10878 catalog_manager.cc:1347] Generated new cluster ID: c98c9b3d89874532a0ac722abdf24b0b
I20260620 12:18:11.653964 10878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:11.674724 10878 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:11.676396 10878 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:11.682415 10878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9: Generated new TSK 0
I20260620 12:18:11.683490 10878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:18:11.722759 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40703
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:11.856161 10884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:11.856552 10884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:11.856623 10884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:11.861128 10884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:11.861202 10884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:11.861291 10884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:18:11.867434 10884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:11.869052 10884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:11.870421 10884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:11.882632 10889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:11.882759 10892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:11.882632 10890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:11.883016 10884 server_base.cc:1034] running on GCE node
I20260620 12:18:11.883687 10884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:11.884542 10884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:11.885738 10884 hybrid_clock.cc:648] HybridClock initialized: now 1781957891885723 us; error 41 us; skew 500 ppm
Jun 20 12:18:11 dist-test-slave-rh5v krb5kdc[10777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957891, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:11.889612 10884 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:18:11.890933 10884 webserver.cc:460] Webserver started at http://127.24.141.1:43875/ using document root <none> and password file <none>
I20260620 12:18:11.891673 10884 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:11.891746 10884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:11.891968 10884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:11.894811 10884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "832a5c07d2fa4d78a577c06ad6a0b379"
format_stamp: "Formatted at 2026-06-20 12:18:11 on dist-test-slave-rh5v"
server_key: "bf24754350272bb60eee584ee397d34e"
server_key_iv: "e2081ea4d4fe80f81758f2e19882e48e"
server_key_version: "encryptionkey@0"
I20260620 12:18:11.895762 10884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "832a5c07d2fa4d78a577c06ad6a0b379"
format_stamp: "Formatted at 2026-06-20 12:18:11 on dist-test-slave-rh5v"
server_key: "bf24754350272bb60eee584ee397d34e"
server_key_iv: "e2081ea4d4fe80f81758f2e19882e48e"
server_key_version: "encryptionkey@0"
I20260620 12:18:11.901302 10884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:18:11.905087 10899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:11.907053 10884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:11.907226 10884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "832a5c07d2fa4d78a577c06ad6a0b379"
format_stamp: "Formatted at 2026-06-20 12:18:11 on dist-test-slave-rh5v"
server_key: "bf24754350272bb60eee584ee397d34e"
server_key_iv: "e2081ea4d4fe80f81758f2e19882e48e"
server_key_version: "encryptionkey@0"
I20260620 12:18:11.907383 10884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:11.933442 10884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:11.937430 10884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:11.937656 10884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:11.938481 10884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:11.939648 10884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:11.939713 10884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:11.939766 10884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:11.939800 10884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:11.951562 10884 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:34777
I20260620 12:18:11.951598 11012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:34777 every 8 connection(s)
I20260620 12:18:11.952826 10884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:18:11.963056 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 10884
I20260620 12:18:11.963236 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:18:11.963574 25140 external_mini_cluster.cc:1428] Setting key 950e5f697a0d019c24c47264c9bdf964
Jun 20 12:18:11 dist-test-slave-rh5v krb5kdc[10777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957891, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:11.971232 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:11.955358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:60321 (local address 127.24.141.62:40703)
0620 12:18:11.955541 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:11.955547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:11.956876 (+ 1329us) server_negotiation.cc:407] Connection header received
0620 12:18:11.958421 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:11.958424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:11.958492 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:11.958614 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:11.960198 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:11.960856 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:11.962357 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:11.962675 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:11.962805 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:11.966157 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:11.966161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:11.966164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:11.966198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:11.968037 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:11.968961 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:11.968965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:11.968967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:11.969220 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:11.969617 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:11.969621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:11.969623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:11.969894 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:11.970091 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:11.970774 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 12:18:11.971029 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:18:11.972186 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:11.955885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40703 (local address 127.24.141.1:60321)
0620 12:18:11.956642 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:11.956710 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:11.958143 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:11.958895 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:11.958904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:11.959372 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:11.959984 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:11.960003 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:11.961055 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:11.961094 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:11.962182 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:11.962200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:11.962930 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:11.962934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:11.963149 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:11.963846 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:11.963867 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:11.965874 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:11.968228 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:11.968239 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:11.968268 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:11.968757 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:11.969332 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:11.969335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:11.969337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:11.969511 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:11.970000 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:11.970007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:11.970389 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 12:18:11.970601 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 12:18:11.970997 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":175,"threads_started":1}
I20260620 12:18:11.973865 11013 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40703
I20260620 12:18:11.974303 11013 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:11.975185 11013 heartbeater.cc:511] Master 127.24.141.62:40703 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:18:11.977404 10822 ts_manager.cc:194] Registered new tserver with Master: 832a5c07d2fa4d78a577c06ad6a0b379 (127.24.141.1:34777)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:18:11.978859 10822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:60321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:18:11.996554 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40703
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:12.129009 11018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:12.129391 11018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:12.129457 11018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:12.133948 11018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:12.134044 11018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:12.134135 11018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:18:12.140569 11018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:12.142019 11018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:12.143201 11018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:12.156780 11023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:12.156770 11024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:12.156831 11018 server_base.cc:1034] running on GCE node
W20260620 12:18:12.156770 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:12.157712 11018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:12.158586 11018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:12.159788 11018 hybrid_clock.cc:648] HybridClock initialized: now 1781957892159764 us; error 39 us; skew 500 ppm
Jun 20 12:18:12 dist-test-slave-rh5v krb5kdc[10777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957892, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:12.163398 11018 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:18:12.164621 11018 webserver.cc:460] Webserver started at http://127.24.141.2:46005/ using document root <none> and password file <none>
I20260620 12:18:12.165376 11018 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:12.165460 11018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:12.165710 11018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:12.168684 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "5b6441462d2543bdb9523a00d244aab6"
format_stamp: "Formatted at 2026-06-20 12:18:12 on dist-test-slave-rh5v"
server_key: "9f968efdb81b5e9ab160cd37f4faa26d"
server_key_iv: "9ec3b65ebdbbe108c8a9b733823d6d17"
server_key_version: "encryptionkey@0"
I20260620 12:18:12.169441 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "5b6441462d2543bdb9523a00d244aab6"
format_stamp: "Formatted at 2026-06-20 12:18:12 on dist-test-slave-rh5v"
server_key: "9f968efdb81b5e9ab160cd37f4faa26d"
server_key_iv: "9ec3b65ebdbbe108c8a9b733823d6d17"
server_key_version: "encryptionkey@0"
I20260620 12:18:12.174485 11018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:12.178045 11033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:12.179667 11018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:18:12.179812 11018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "5b6441462d2543bdb9523a00d244aab6"
format_stamp: "Formatted at 2026-06-20 12:18:12 on dist-test-slave-rh5v"
server_key: "9f968efdb81b5e9ab160cd37f4faa26d"
server_key_iv: "9ec3b65ebdbbe108c8a9b733823d6d17"
server_key_version: "encryptionkey@0"
I20260620 12:18:12.179944 11018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:12.194303 11018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:12.199182 11018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:12.199453 11018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:12.200273 11018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:12.201428 11018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:12.201483 11018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:12.201545 11018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:12.201571 11018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:12.213222 11018 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:42613
I20260620 12:18:12.213263 11146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:42613 every 8 connection(s)
I20260620 12:18:12.214466 11018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:18:12.215453 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11018
I20260620 12:18:12.215580 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:18:12.215907 25140 external_mini_cluster.cc:1428] Setting key b5bca4d7923174b09b4ae71dded08847
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
Jun 20 12:18:12 dist-test-slave-rh5v krb5kdc[10777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957892, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:12.236073 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.216908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:49589 (local address 127.24.141.62:40703)
0620 12:18:12.217136 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.217141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.218919 (+ 1778us) server_negotiation.cc:407] Connection header received
0620 12:18:12.220338 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.220342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.220423 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.220547 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:12.222052 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.222891 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.224827 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.225142 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.225282 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.231759 (+ 6477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:12.231763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:12.231765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:12.231799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:12.233718 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:12.234366 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:12.234369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:12.234371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:12.234452 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:12.234830 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:12.234833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:12.234834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:12.235049 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:12.235140 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.235741 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.235893 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:18:12.237032 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.217453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40703 (local address 127.24.141.2:49589)
0620 12:18:12.218711 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:12.218762 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:12.220068 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:12.220756 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:12.220765 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:12.221239 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:12.221839 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.221857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.223146 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.223151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.224547 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.224574 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.225358 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.225363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.225596 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.226480 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:12.226508 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:12.231488 (+ 4980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:12.233889 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:12.233898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:12.233919 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:12.234239 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:12.234543 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:12.234546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:12.234548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:12.234722 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:12.235160 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:12.235169 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:12.235473 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 12:18:12.235660 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 12:18:12.236061 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":194,"threads_started":1}
I20260620 12:18:12.239301 11147 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40703
I20260620 12:18:12.239671 11147 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:12.240520 11147 heartbeater.cc:511] Master 127.24.141.62:40703 requested a full tablet report, sending...
I20260620 12:18:12.242277 10822 ts_manager.cc:194] Registered new tserver with Master: 5b6441462d2543bdb9523a00d244aab6 (127.24.141.2:42613)
I20260620 12:18:12.243008 10822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:49589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:18:12.249189 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40703
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:12.384658 11152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:12.385026 11152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:12.385154 11152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:12.390589 11152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:12.390707 11152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:12.390811 11152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:18:12.399067 11152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:12.401134 11152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:12.402886 11152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:12.416216 11158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:12.416198 11160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:12.416718 11152 server_base.cc:1034] running on GCE node
W20260620 12:18:12.416743 11157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:12.417349 11152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:12.418184 11152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:12.419510 11152 hybrid_clock.cc:648] HybridClock initialized: now 1781957892419465 us; error 71 us; skew 500 ppm
Jun 20 12:18:12 dist-test-slave-rh5v krb5kdc[10777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957892, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:12.423636 11152 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:18:12.424865 11152 webserver.cc:460] Webserver started at http://127.24.141.3:35285/ using document root <none> and password file <none>
I20260620 12:18:12.425664 11152 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:12.425735 11152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:12.425972 11152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:12.428674 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "e5ced60fe24d4b15aeaf34538cf71064"
format_stamp: "Formatted at 2026-06-20 12:18:12 on dist-test-slave-rh5v"
server_key: "5d32f6561318613e756090b7f20b37c7"
server_key_iv: "a33cc98876ba95cd57a3813bcec50c63"
server_key_version: "encryptionkey@0"
I20260620 12:18:12.429639 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "e5ced60fe24d4b15aeaf34538cf71064"
format_stamp: "Formatted at 2026-06-20 12:18:12 on dist-test-slave-rh5v"
server_key: "5d32f6561318613e756090b7f20b37c7"
server_key_iv: "a33cc98876ba95cd57a3813bcec50c63"
server_key_version: "encryptionkey@0"
I20260620 12:18:12.434900 11152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:12.438357 11167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:12.440227 11152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:12.440402 11152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "e5ced60fe24d4b15aeaf34538cf71064"
format_stamp: "Formatted at 2026-06-20 12:18:12 on dist-test-slave-rh5v"
server_key: "5d32f6561318613e756090b7f20b37c7"
server_key_iv: "a33cc98876ba95cd57a3813bcec50c63"
server_key_version: "encryptionkey@0"
I20260620 12:18:12.440593 11152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:12.460881 11152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:12.465040 11152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:12.465287 11152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:12.466063 11152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:12.467105 11152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:12.467163 11152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:12.467208 11152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:12.467223 11152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:12.479070 11152 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:40117
I20260620 12:18:12.479108 11280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:40117 every 8 connection(s)
I20260620 12:18:12.480221 11152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:18:12.488279 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11152
I20260620 12:18:12.488448 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:18:12.488807 25140 external_mini_cluster.cc:1428] Setting key 7718dc7c39324b145f4aba9dd8211ded
Jun 20 12:18:12 dist-test-slave-rh5v krb5kdc[10777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957892, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:12.498886 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.482590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:40225 (local address 127.24.141.62:40703)
0620 12:18:12.482757 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.482762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.484121 (+ 1359us) server_negotiation.cc:407] Connection header received
0620 12:18:12.485344 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.485348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.485416 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.485494 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:12.487031 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.487636 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.489331 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.489563 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.489671 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.494051 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:12.494055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:12.494058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:12.494096 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:12.495671 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:12.496567 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:12.496570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:12.496572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:12.496676 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:12.497263 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:12.497266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:12.497268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:12.497574 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:12.497674 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.498493 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.498672 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:18:12.500068 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.482926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:40703 (local address 127.24.141.3:40225)
0620 12:18:12.483913 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:12.483973 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:12.485044 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:12.485742 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:12.485752 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:12.486216 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:12.486867 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.486884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.487764 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.487767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.489194 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.489208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.489857 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.489866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.490236 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.491145 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:12.491174 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:12.493647 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:12.495892 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:12.495900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:12.495920 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:12.496343 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:12.496828 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:12.496831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:12.496833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:12.497043 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:12.497749 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:12.497759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:12.498169 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 12:18:12.498399 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 12:18:12.499038 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":163,"threads_started":1}
I20260620 12:18:12.501950 11281 heartbeater.cc:347] Connected to a master server at 127.24.141.62:40703
I20260620 12:18:12.502297 11281 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:12.503077 11281 heartbeater.cc:511] Master 127.24.141.62:40703 requested a full tablet report, sending...
I20260620 12:18:12.504415 10822 ts_manager.cc:194] Registered new tserver with Master: e5ced60fe24d4b15aeaf34538cf71064 (127.24.141.3:40117)
I20260620 12:18:12.505210 10822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:40225
I20260620 12:18:12.516779 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:18:12.528407 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.519628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.24.141.62:40703)
0620 12:18:12.519782 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.519786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.519804 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:18:12.519953 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.519956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.520009 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.520112 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:12.520946 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.521561 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.522519 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.522786 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.522902 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.524654 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:12.524658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:12.524660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:12.524690 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:12.526343 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:12.527029 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:12.527033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:12.527034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:12.527148 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:12.527462 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:12.527464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:12.527465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:12.527697 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:12.527824 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.528139 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.528257 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:18:12.532155 10822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:18:12.536155 10822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:18:12.553772 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.547172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:40117 (local address 127.0.0.1:60140)
0620 12:18:12.548115 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:12.548132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:12.548222 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:12.549240 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:12.549243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:12.549264 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:12.549401 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.549409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.550567 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.550570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.551907 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.551919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.553163 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.553166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.553267 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.553301 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:18:12.553420 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 12:18:12.553578 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":144,"threads_started":1}
I20260620 12:18:12.554107 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.547189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.24.141.1:34777)
0620 12:18:12.548504 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.548512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.548540 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:18:12.548633 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.548638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.548883 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.549496 (+ 613us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:12.549652 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.550860 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.552362 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.553305 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.553453 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.553622 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.553683 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.553799 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":152,"threads_started":1}
I20260620 12:18:12.554220 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.547926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.24.141.3:40117)
0620 12:18:12.548666 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.548678 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.548700 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:18:12.548776 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.548781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.549028 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.549272 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:12.549530 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.550421 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.552054 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.552777 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.552883 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.553100 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.553757 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.553911 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":172,"threads_started":1}
I20260620 12:18:12.554327 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.547907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:42613 (local address 127.0.0.1:42188)
0620 12:18:12.548270 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:12.548289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:12.548416 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:12.549510 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:12.549512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:12.549529 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:12.549608 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.549614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.551059 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.551062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.552245 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.552256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.553831 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.553835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.553919 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.553940 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:18:12.554010 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 12:18:12.554095 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
I20260620 12:18:12.553777 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.547011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34777 (local address 127.0.0.1:53400)
0620 12:18:12.547511 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:12.547566 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:12.547714 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:12.549104 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:12.549111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:12.549138 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:12.549325 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.549332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.551009 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.551013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.552225 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.552234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.553446 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.553450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.553533 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.553554 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:18:12.553621 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 12:18:12.553687 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":7,"thread_start_us":99,"threads_started":1}
I20260620 12:18:12.554652 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.548281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.24.141.2:42613)
0620 12:18:12.549013 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.549026 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.549048 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:18:12.549153 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.549159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.549355 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.549503 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:12.549734 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.550874 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.552399 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.553305 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.553453 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.553622 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.554405 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.554518 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":141,"threads_started":1}
I20260620 12:18:12.558070 10947 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd6490d2833443eafffd165b499c107 (DEFAULT_TABLE table=test-table [id=75b18f1598bd46408120cd4e2c5653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:12.558115 11215 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd6490d2833443eafffd165b499c107 (DEFAULT_TABLE table=test-table [id=75b18f1598bd46408120cd4e2c5653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:12.558115 11081 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd6490d2833443eafffd165b499c107 (DEFAULT_TABLE table=test-table [id=75b18f1598bd46408120cd4e2c5653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:12.559374 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd6490d2833443eafffd165b499c107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:12.559485 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd6490d2833443eafffd165b499c107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:12.559485 11081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd6490d2833443eafffd165b499c107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:12.567070 11298 tablet_bootstrap.cc:492] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379: Bootstrap starting.
I20260620 12:18:12.569507 11299 tablet_bootstrap.cc:492] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Bootstrap starting.
I20260620 12:18:12.569679 11300 tablet_bootstrap.cc:492] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Bootstrap starting.
I20260620 12:18:12.569885 11298 tablet_bootstrap.cc:654] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:12.571169 11298 log.cc:824] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:12.572319 11300 tablet_bootstrap.cc:654] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:12.572716 11299 tablet_bootstrap.cc:654] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:12.573637 11298 tablet_bootstrap.cc:492] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379: No bootstrap required, opened a new log
I20260620 12:18:12.573822 11300 log.cc:824] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:12.573822 11299 log.cc:824] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:12.573890 11298 ts_tablet_manager.cc:1397] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:18:12.576346 11300 tablet_bootstrap.cc:492] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: No bootstrap required, opened a new log
I20260620 12:18:12.576346 11299 tablet_bootstrap.cc:492] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: No bootstrap required, opened a new log
I20260620 12:18:12.576589 11299 ts_tablet_manager.cc:1397] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 12:18:12.576589 11300 ts_tablet_manager.cc:1397] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 12:18:12.578395 11298 raft_consensus.cc:348] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.578716 11298 raft_consensus.cc:374] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:12.578783 11298 raft_consensus.cc:729] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832a5c07d2fa4d78a577c06ad6a0b379, State: Initialized, Role: FOLLOWER
I20260620 12:18:12.579316 11298 consensus_queue.cc:260] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.580139 11013 heartbeater.cc:503] Master 127.24.141.62:40703 was elected leader, sending a full tablet report...
I20260620 12:18:12.580631 11298 ts_tablet_manager.cc:1428] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:18:12.580857 11300 raft_consensus.cc:348] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.580857 11299 raft_consensus.cc:348] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.581283 11300 raft_consensus.cc:374] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:12.581283 11299 raft_consensus.cc:374] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:12.581354 11299 raft_consensus.cc:729] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6441462d2543bdb9523a00d244aab6, State: Initialized, Role: FOLLOWER
I20260620 12:18:12.581354 11300 raft_consensus.cc:729] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ced60fe24d4b15aeaf34538cf71064, State: Initialized, Role: FOLLOWER
I20260620 12:18:12.581928 11299 consensus_queue.cc:260] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.581928 11300 consensus_queue.cc:260] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.583190 11281 heartbeater.cc:503] Master 127.24.141.62:40703 was elected leader, sending a full tablet report...
I20260620 12:18:12.583230 11299 ts_tablet_manager.cc:1428] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 12:18:12.583235 11300 ts_tablet_manager.cc:1428] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 12:18:12.583806 11147 heartbeater.cc:503] Master 127.24.141.62:40703 was elected leader, sending a full tablet report...
W20260620 12:18:12.704710 11014 tablet.cc:2369] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:18:12.715991 11148 tablet.cc:2369] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:18:12.731593 11282 tablet.cc:2369] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:12.941965 11304 raft_consensus.cc:482] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:18:12.942248 11304 raft_consensus.cc:504] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.943782 11304 leader_election.cc:288] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5ced60fe24d4b15aeaf34538cf71064 (127.24.141.3:40117), 5b6441462d2543bdb9523a00d244aab6 (127.24.141.2:42613)
I20260620 12:18:12.950179 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.944313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:44173 (local address 127.24.141.2:42613)
0620 12:18:12.944605 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.944610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.944844 (+ 234us) server_negotiation.cc:407] Connection header received
0620 12:18:12.944965 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.944968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.945033 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.945162 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:12.945712 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.946531 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.948575 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.949296 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.949472 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.949531 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.949918 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.950021 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260620 12:18:12.950179 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.944222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:45653 (local address 127.24.141.3:40117)
0620 12:18:12.944416 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:12.944421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:12.944773 (+ 352us) server_negotiation.cc:407] Connection header received
0620 12:18:12.944971 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:12.944974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:12.945032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:12.945156 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:12.945712 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.946531 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.948575 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.949296 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.949472 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.949531 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:12.949918 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 12:18:12.950020 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 12:18:12.950237 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.944086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:40117 (local address 127.24.141.1:45653)
0620 12:18:12.944623 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:12.944642 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:12.944811 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:12.945176 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:12.945179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:12.945225 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:12.945532 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.945541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.946708 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.946712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.948350 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.948364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.949501 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.949505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.949693 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.949749 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 12:18:12.949844 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 12:18:12.949995 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":232,"spinlock_wait_cycles":6656,"thread_start_us":172,"threads_started":1}
I20260620 12:18:12.950237 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:12.944222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:42613 (local address 127.24.141.1:44173)
0620 12:18:12.944762 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:12.944775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:12.944861 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:12.945176 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:12.945179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:12.945223 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:12.945547 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.945553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.946708 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.946712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.948350 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:12.948364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:12.949501 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:12.949505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:12.949699 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:12.949753 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 12:18:12.949844 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:18:12.950000 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":242,"spinlock_wait_cycles":44288,"thread_start_us":150,"threads_started":1}
I20260620 12:18:12.951166 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd6490d2833443eafffd165b499c107" candidate_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6441462d2543bdb9523a00d244aab6" is_pre_election: true
I20260620 12:18:12.951165 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd6490d2833443eafffd165b499c107" candidate_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ced60fe24d4b15aeaf34538cf71064" is_pre_election: true
I20260620 12:18:12.951618 11101 raft_consensus.cc:2454] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 832a5c07d2fa4d78a577c06ad6a0b379 in term 0.
I20260620 12:18:12.951618 11235 raft_consensus.cc:2454] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 832a5c07d2fa4d78a577c06ad6a0b379 in term 0.
I20260620 12:18:12.952256 10903 leader_election.cc:302] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b6441462d2543bdb9523a00d244aab6, 832a5c07d2fa4d78a577c06ad6a0b379, e5ced60fe24d4b15aeaf34538cf71064; no voters:
I20260620 12:18:12.952570 11304 raft_consensus.cc:2781] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:18:12.952659 11304 raft_consensus.cc:482] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:18:12.952706 11304 raft_consensus.cc:3037] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:12.954129 11304 raft_consensus.cc:504] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.954557 11304 leader_election.cc:288] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [CANDIDATE]: Term 1 election: Requested vote from peers e5ced60fe24d4b15aeaf34538cf71064 (127.24.141.3:40117), 5b6441462d2543bdb9523a00d244aab6 (127.24.141.2:42613)
I20260620 12:18:12.954984 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd6490d2833443eafffd165b499c107" candidate_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ced60fe24d4b15aeaf34538cf71064"
I20260620 12:18:12.955114 11101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd6490d2833443eafffd165b499c107" candidate_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6441462d2543bdb9523a00d244aab6"
I20260620 12:18:12.955142 11235 raft_consensus.cc:3037] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:12.955258 11101 raft_consensus.cc:3037] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:12.956681 11101 raft_consensus.cc:2454] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 832a5c07d2fa4d78a577c06ad6a0b379 in term 1.
I20260620 12:18:12.956681 11235 raft_consensus.cc:2454] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 832a5c07d2fa4d78a577c06ad6a0b379 in term 1.
I20260620 12:18:12.957108 10901 leader_election.cc:302] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6441462d2543bdb9523a00d244aab6, 832a5c07d2fa4d78a577c06ad6a0b379; no voters:
I20260620 12:18:12.957314 11304 raft_consensus.cc:2781] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:12.957602 11304 raft_consensus.cc:686] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [term 1 LEADER]: Becoming Leader. State: Replica: 832a5c07d2fa4d78a577c06ad6a0b379, State: Running, Role: LEADER
I20260620 12:18:12.958060 11304 consensus_queue.cc:237] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } }
I20260620 12:18:12.961838 10822 catalog_manager.cc:5495] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 reported cstate change: term changed from 0 to 1, leader changed from <none> to 832a5c07d2fa4d78a577c06ad6a0b379 (127.24.141.1). New cstate: current_term: 1 leader_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832a5c07d2fa4d78a577c06ad6a0b379" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 34777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 } health_report { overall_health: UNKNOWN } } }
I20260620 12:18:13.012460 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:13.007996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.24.141.1:34777)
0620 12:18:13.008169 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:13.008174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:13.008191 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:13.008331 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:13.008334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:13.008401 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:13.008512 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:13.008792 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.009441 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.010446 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.010674 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.010764 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:13.011235 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:13.011846 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:13.011999 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:13.012206 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 12:18:13.012261 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:18:13.021277 11235 raft_consensus.cc:1261] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 1 FOLLOWER]: Refusing update from remote peer 832a5c07d2fa4d78a577c06ad6a0b379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:13.021277 11101 raft_consensus.cc:1261] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 1 FOLLOWER]: Refusing update from remote peer 832a5c07d2fa4d78a577c06ad6a0b379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:13.022467 11309 consensus_queue.cc:1035] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5ced60fe24d4b15aeaf34538cf71064" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:13.022820 11304 consensus_queue.cc:1035] T 2fd6490d2833443eafffd165b499c107 P 832a5c07d2fa4d78a577c06ad6a0b379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b6441462d2543bdb9523a00d244aab6" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:13.032774 11312 mvcc.cc:205] Tried to move back new op lower bound from 7298899529803407360 to 7298899529559470080. Current Snapshot: MvccSnapshot[applied={T|T < 7298899529803407360}]
I20260620 12:18:13.044885 10822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:13.045157 10822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:13.048593 10822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e59636190217440881bc9562d613d2c9: Sending DeleteTablet for 3 replicas of tablet 2fd6490d2833443eafffd165b499c107
I20260620 12:18:13.049873 11215 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd6490d2833443eafffd165b499c107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:13 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:60140
I20260620 12:18:13.049887 10947 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd6490d2833443eafffd165b499c107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:13 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:53400
I20260620 12:18:13.050329 11081 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd6490d2833443eafffd165b499c107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:13 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:42188
I20260620 12:18:13.050596 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 10884
I20260620 12:18:13.050631 11322 tablet_replica.cc:330] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: stopping tablet replica
I20260620 12:18:13.051187 11322 raft_consensus.cc:2229] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:13.051680 11322 raft_consensus.cc:2258] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:13.051672 11324 tablet_replica.cc:330] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: stopping tablet replica
I20260620 12:18:13.052601 11324 raft_consensus.cc:2229] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:13.053092 11324 raft_consensus.cc:2258] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:13.053911 11322 ts_tablet_manager.cc:1905] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:13.054917 11324 ts_tablet_manager.cc:1905] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:13.057510 11322 ts_tablet_manager.cc:1918] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:13.057648 11322 log.cc:1196] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/2fd6490d2833443eafffd165b499c107
I20260620 12:18:13.057713 11324 ts_tablet_manager.cc:1918] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:13.057852 11324 log.cc:1196] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/2fd6490d2833443eafffd165b499c107
I20260620 12:18:13.057994 11322 ts_tablet_manager.cc:1939] T 2fd6490d2833443eafffd165b499c107 P e5ced60fe24d4b15aeaf34538cf71064: Deleting consensus metadata
I20260620 12:18:13.058203 11324 ts_tablet_manager.cc:1939] T 2fd6490d2833443eafffd165b499c107 P 5b6441462d2543bdb9523a00d244aab6: Deleting consensus metadata
I20260620 12:18:13.059659 10807 catalog_manager.cc:4841] TS 5b6441462d2543bdb9523a00d244aab6 (127.24.141.2:42613): tablet 2fd6490d2833443eafffd165b499c107 (table test-table [id=75b18f1598bd46408120cd4e2c5653c2]) successfully deleted
I20260620 12:18:13.059681 10809 catalog_manager.cc:4841] TS e5ced60fe24d4b15aeaf34538cf71064 (127.24.141.3:40117): tablet 2fd6490d2833443eafffd165b499c107 (table test-table [id=75b18f1598bd46408120cd4e2c5653c2]) successfully deleted
I20260620 12:18:13.061077 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11018
W20260620 12:18:13.061131 10807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:34777 (error 108)
I20260620 12:18:13.063856 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:13.063355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:34777 (local address 127.0.0.1:53426)
0620 12:18:13.063610 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:13.063726 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:34777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
W20260620 12:18:13.064064 10807 catalog_manager.cc:4584] TS 832a5c07d2fa4d78a577c06ad6a0b379 (127.24.141.1:34777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fd6490d2833443eafffd165b499c107: Network error: Client connection negotiation failed: client connection to 127.24.141.1:34777: connect: Connection refused (error 111)
I20260620 12:18:13.068167 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11152
I20260620 12:18:13.075310 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 10790
2026-06-20T12:18:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T12:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:13Z Disabled control of system clock
I20260620 12:18:13.110285 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:41639
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:41639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:18:13.242084 11330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:13.242486 11330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:13.242532 11330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:13.246999 11330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:13.247092 11330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:13.247107 11330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:13.247123 11330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:13.247138 11330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:13.252614 11330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:41639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:13.254170 11330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:13.255916 11330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:13.266250 11335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:13.266203 11336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:13.266215 11338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:13.266536 11330 server_base.cc:1034] running on GCE node
I20260620 12:18:13.267416 11330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:13.268829 11330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:13.270087 11330 hybrid_clock.cc:648] HybridClock initialized: now 1781957893270055 us; error 46 us; skew 500 ppm
I20260620 12:18:13.276500 11330 webserver.cc:460] Webserver started at http://127.24.141.62:44673/ using document root <none> and password file <none>
I20260620 12:18:13.277472 11330 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:13.277577 11330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:13.277869 11330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:13.280475 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "76b3ff8090f44bc4ad144ab876e0519a"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "40d4034b034c4217d07770c0037606e3"
server_key_iv: "6114c01f12a0ab10edf2d294992da705"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.281255 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "76b3ff8090f44bc4ad144ab876e0519a"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "40d4034b034c4217d07770c0037606e3"
server_key_iv: "6114c01f12a0ab10edf2d294992da705"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.287191 11330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 12:18:13.290817 11344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:13.292716 11330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 12:18:13.292945 11330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "76b3ff8090f44bc4ad144ab876e0519a"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "40d4034b034c4217d07770c0037606e3"
server_key_iv: "6114c01f12a0ab10edf2d294992da705"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.293144 11330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:13.314435 11330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:13.315352 11330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:13.315577 11330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:13.323771 11330 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:41639
I20260620 12:18:13.323807 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:41639 every 8 connection(s)
I20260620 12:18:13.324958 11330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:13.329315 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:13.329357 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11330
I20260620 12:18:13.329479 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:13.329869 25140 external_mini_cluster.cc:1428] Setting key 6afe29612966683dfa5d5aea295c2cc9
I20260620 12:18:13.340334 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a: Bootstrap starting.
I20260620 12:18:13.342847 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:13.331985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.24.141.62:41639)
0620 12:18:13.333051 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:13.333111 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:13.333168 (+ 57us) server_negotiation.cc:407] Connection header received
0620 12:18:13.334265 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:13.334293 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:13.334860 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:13.335280 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:13.335862 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.336991 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.338419 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.339162 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.339348 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:13.340562 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:13.340571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:13.340595 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:13.341154 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:13.341238 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:13.341266 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:13.341369 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:13.341542 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 12:18:13.341925 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":193,"threads_started":1}
I20260620 12:18:13.343622 11397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:13.344587 11397 log.cc:824] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:13.347220 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a: No bootstrap required, opened a new log
I20260620 12:18:13.351333 11397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3ff8090f44bc4ad144ab876e0519a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41639 } }
I20260620 12:18:13.351701 11397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:13.351758 11397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b3ff8090f44bc4ad144ab876e0519a, State: Initialized, Role: FOLLOWER
I20260620 12:18:13.352319 11397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3ff8090f44bc4ad144ab876e0519a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41639 } }
I20260620 12:18:13.352475 11397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:13.352546 11397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:13.352653 11397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:13.353886 11397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3ff8090f44bc4ad144ab876e0519a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41639 } }
I20260620 12:18:13.354285 11397 leader_election.cc:302] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b3ff8090f44bc4ad144ab876e0519a; no voters:
I20260620 12:18:13.354647 11397 leader_election.cc:288] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:13.354905 11402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:13.355264 11402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [term 1 LEADER]: Becoming Leader. State: Replica: 76b3ff8090f44bc4ad144ab876e0519a, State: Running, Role: LEADER
I20260620 12:18:13.355679 11402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3ff8090f44bc4ad144ab876e0519a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41639 } }
I20260620 12:18:13.356168 11397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:13.357627 11404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b3ff8090f44bc4ad144ab876e0519a. Latest consensus state: current_term: 1 leader_uuid: "76b3ff8090f44bc4ad144ab876e0519a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3ff8090f44bc4ad144ab876e0519a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41639 } } }
I20260620 12:18:13.357781 11404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:13.358271 11403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b3ff8090f44bc4ad144ab876e0519a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b3ff8090f44bc4ad144ab876e0519a" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41639 } } }
I20260620 12:18:13.358453 11403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a [sys.catalog]: This master's current role is: LEADER
W20260620 12:18:13.362437 11417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:13.362552 11417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:13.362659 11412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:13.363597 11412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:13.370888 11412 catalog_manager.cc:1347] Generated new cluster ID: 01b68bffa0a54c12a6c60d0ce4a8546c
I20260620 12:18:13.371001 11412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:13.398170 11412 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:13.399267 11412 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:13.406975 11412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a: Generated new TSK 0
I20260620 12:18:13.407989 11412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:13.426050 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:41639
--builtin_ntp_servers=127.24.141.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:18:13.558701 11421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:13.559029 11421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:13.559070 11421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:13.563431 11421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:13.563519 11421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:13.563578 11421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:18:13.568868 11421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:13.570158 11421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:13.571695 11421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:13.583022 11426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:13.583019 11427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:13.583220 11421 server_base.cc:1034] running on GCE node
W20260620 12:18:13.583487 11429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:13.584070 11421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:13.584915 11421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:13.586246 11421 hybrid_clock.cc:648] HybridClock initialized: now 1781957893586214 us; error 66 us; skew 500 ppm
I20260620 12:18:13.588789 11421 webserver.cc:460] Webserver started at http://127.24.141.1:44435/ using document root <none> and password file <none>
I20260620 12:18:13.589609 11421 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:13.589694 11421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:13.589916 11421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:13.592648 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "15bdff712cc5474e8e78deea69acf0ca"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "ad64c99b07256b0f9576fb002a4311f1"
server_key_iv: "c6a41f44c5fdc8579e19cc6fba7eaa93"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.593437 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "15bdff712cc5474e8e78deea69acf0ca"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "ad64c99b07256b0f9576fb002a4311f1"
server_key_iv: "c6a41f44c5fdc8579e19cc6fba7eaa93"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.598615 11421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:18:13.601919 11435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:13.603598 11421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:18:13.603744 11421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "15bdff712cc5474e8e78deea69acf0ca"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "ad64c99b07256b0f9576fb002a4311f1"
server_key_iv: "c6a41f44c5fdc8579e19cc6fba7eaa93"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.603864 11421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:13.623111 11421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:13.624033 11421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:13.624218 11421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:13.624958 11421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:13.626240 11421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:13.626299 11421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:13.626363 11421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:13.626392 11421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:13.637326 11421 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:45355
I20260620 12:18:13.637382 11548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:45355 every 8 connection(s)
I20260620 12:18:13.638497 11421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:18:13.646646 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11421
I20260620 12:18:13.646799 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:18:13.647143 25140 external_mini_cluster.cc:1428] Setting key 874ee3b12d0f4125bf5cd12a00693bdb
I20260620 12:18:13.649678 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:41639
--builtin_ntp_servers=127.24.141.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:18:13.650362 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:13.640998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:33895 (local address 127.24.141.62:41639)
0620 12:18:13.641166 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:13.641170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:13.642809 (+ 1639us) server_negotiation.cc:407] Connection header received
0620 12:18:13.644429 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:13.644432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:13.644507 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:13.644588 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:13.645715 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.646287 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.647596 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.647836 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.647958 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:13.649653 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:13.649657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:13.649659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:13.649675 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:13.649695 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:13.649703 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:13.649777 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:13.650090 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 12:18:13.650193 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 12:18:13.651345 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:13.641579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:41639 (local address 127.24.141.1:33895)
0620 12:18:13.642589 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:13.642651 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:13.644135 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:13.644817 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:13.644826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:13.644930 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:13.645555 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:13.645572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.646429 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.646432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:13.647465 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:13.647478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.648077 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.648083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:13.648376 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:13.649314 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:13.649316 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:13.649374 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:13.649470 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:13.649927 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:13.649938 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:13.649945 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:18:13.650066 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 12:18:13.650439 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":158,"threads_started":1}
I20260620 12:18:13.653196 11549 heartbeater.cc:347] Connected to a master server at 127.24.141.62:41639
I20260620 12:18:13.653539 11549 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:13.654198 11549 heartbeater.cc:511] Master 127.24.141.62:41639 requested a full tablet report, sending...
I20260620 12:18:13.656601 11361 ts_manager.cc:194] Registered new tserver with Master: 15bdff712cc5474e8e78deea69acf0ca (127.24.141.1:45355)
I20260620 12:18:13.658679 11361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.1:33895
W20260620 12:18:13.786540 11552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:13.786911 11552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:13.786955 11552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:13.791452 11552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:13.791531 11552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:13.791599 11552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:18:13.796928 11552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:13.798539 11552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:13.800376 11552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:13.812371 11557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:13.812371 11558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:13.812490 11560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:13.812935 11552 server_base.cc:1034] running on GCE node
I20260620 12:18:13.813506 11552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:13.814356 11552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:13.815573 11552 hybrid_clock.cc:648] HybridClock initialized: now 1781957893815548 us; error 50 us; skew 500 ppm
I20260620 12:18:13.818215 11552 webserver.cc:460] Webserver started at http://127.24.141.2:36455/ using document root <none> and password file <none>
I20260620 12:18:13.818974 11552 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:13.819051 11552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:13.819288 11552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:13.822093 11552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "dfea0e62a31746cfa3718fe4004c50c8"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "dfa273b0253518400e658c49c418aa9c"
server_key_iv: "8e3f7dfb145312bb11735ab3a99693ae"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.822778 11552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "dfea0e62a31746cfa3718fe4004c50c8"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "dfa273b0253518400e658c49c418aa9c"
server_key_iv: "8e3f7dfb145312bb11735ab3a99693ae"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.827673 11552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:13.831099 11566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:13.833246 11552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:13.833395 11552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "dfea0e62a31746cfa3718fe4004c50c8"
format_stamp: "Formatted at 2026-06-20 12:18:13 on dist-test-slave-rh5v"
server_key: "dfa273b0253518400e658c49c418aa9c"
server_key_iv: "8e3f7dfb145312bb11735ab3a99693ae"
server_key_version: "encryptionkey@0"
I20260620 12:18:13.833577 11552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:13.846223 11552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:13.847090 11552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:13.847289 11552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:13.848007 11552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:13.849238 11552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:13.849291 11552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:13.849352 11552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:13.849378 11552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:13.860986 11552 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:45039
I20260620 12:18:13.861006 11679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:45039 every 8 connection(s)
I20260620 12:18:13.862236 11552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:18:13.869395 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11552
I20260620 12:18:13.869581 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:18:13.869923 25140 external_mini_cluster.cc:1428] Setting key f588599a0f1f326a244fa663ee3280b6
I20260620 12:18:13.872529 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:41639
--builtin_ntp_servers=127.24.141.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:18:13.874771 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:13.864902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:57847 (local address 127.24.141.62:41639)
0620 12:18:13.865053 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:13.865090 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:13.866812 (+ 1722us) server_negotiation.cc:407] Connection header received
0620 12:18:13.868445 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:13.868449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:13.868536 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:13.868657 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:13.870196 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.870865 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.872336 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.872589 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.872691 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:13.874165 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:13.874170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:13.874173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:13.874191 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:13.874218 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:13.874228 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:13.874297 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:13.874512 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 12:18:13.874599 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:18:13.875823 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:13.865481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:41639 (local address 127.24.141.2:57847)
0620 12:18:13.866570 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:13.866632 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:13.868128 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:13.868989 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:13.868999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:13.869191 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:13.869925 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:13.869946 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.871035 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.871038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:13.872183 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:13.872198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:13.872771 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:13.872775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:13.872966 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:13.873787 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:13.873789 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:13.873842 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:13.873952 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:13.874339 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:13.874348 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:13.874355 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:18:13.874484 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 12:18:13.874844 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":198,"threads_started":1}
I20260620 12:18:13.877650 11680 heartbeater.cc:347] Connected to a master server at 127.24.141.62:41639
I20260620 12:18:13.878096 11680 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:13.879019 11680 heartbeater.cc:511] Master 127.24.141.62:41639 requested a full tablet report, sending...
I20260620 12:18:13.880859 11361 ts_manager.cc:194] Registered new tserver with Master: dfea0e62a31746cfa3718fe4004c50c8 (127.24.141.2:45039)
I20260620 12:18:13.881649 11361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.2:57847
W20260620 12:18:14.004010 11683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:14.004349 11683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:14.004391 11683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:14.009203 11683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:14.009284 11683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:14.009352 11683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:18:14.014803 11683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:14.016064 11683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:14.017705 11683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:14.029119 11691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:14.029176 11688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:14.029336 11689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:14.029731 11683 server_base.cc:1034] running on GCE node
I20260620 12:18:14.030277 11683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:14.031077 11683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:14.032316 11683 hybrid_clock.cc:648] HybridClock initialized: now 1781957894032275 us; error 61 us; skew 500 ppm
I20260620 12:18:14.034937 11683 webserver.cc:460] Webserver started at http://127.24.141.3:36975/ using document root <none> and password file <none>
I20260620 12:18:14.035667 11683 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:14.035751 11683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:14.035985 11683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:14.038515 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501"
format_stamp: "Formatted at 2026-06-20 12:18:14 on dist-test-slave-rh5v"
server_key: "6b9043225cd6604666063808e957f93a"
server_key_iv: "d9d06b085fa3a68350ac136d86f3b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:18:14.039222 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501"
format_stamp: "Formatted at 2026-06-20 12:18:14 on dist-test-slave-rh5v"
server_key: "6b9043225cd6604666063808e957f93a"
server_key_iv: "d9d06b085fa3a68350ac136d86f3b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:18:14.044962 11683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:18:14.048236 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:14.050143 11683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 12:18:14.050339 11683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501"
format_stamp: "Formatted at 2026-06-20 12:18:14 on dist-test-slave-rh5v"
server_key: "6b9043225cd6604666063808e957f93a"
server_key_iv: "d9d06b085fa3a68350ac136d86f3b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:18:14.050493 11683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:14.070742 11683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:14.071542 11683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:14.071740 11683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:14.072415 11683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:14.073606 11683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:14.073659 11683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:14.073710 11683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:14.073745 11683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:14.084211 11683 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:33347
I20260620 12:18:14.084285 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:33347 every 8 connection(s)
I20260620 12:18:14.085409 11683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:18:14.092061 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11683
I20260620 12:18:14.092234 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:18:14.092598 25140 external_mini_cluster.cc:1428] Setting key 41ba690876fc4a6c4c2c1222c37dd310
I20260620 12:18:14.098599 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.088027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:35413 (local address 127.24.141.62:41639)
0620 12:18:14.088237 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.088242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.089732 (+ 1490us) server_negotiation.cc:407] Connection header received
0620 12:18:14.091507 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.091511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.091595 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.091710 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:14.093385 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.094175 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.096124 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.096375 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.096524 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.097904 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:14.097908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:14.097911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:14.097929 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:14.097966 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:14.097976 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:14.098064 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.098316 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.098410 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:18:14.099542 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.088491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:41639 (local address 127.24.141.3:35413)
0620 12:18:14.089492 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:14.089551 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:14.091188 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:14.092069 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:14.092081 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:14.092233 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:14.093172 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.093195 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.094434 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.094440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.095945 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.095967 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.096580 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.096584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.096768 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.097536 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:14.097538 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:14.097593 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:14.097683 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:14.098084 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:14.098092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:14.098097 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:18:14.098219 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 12:18:14.098517 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":169,"threads_started":1}
I20260620 12:18:14.101277 11811 heartbeater.cc:347] Connected to a master server at 127.24.141.62:41639
I20260620 12:18:14.101722 11811 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:14.102527 11811 heartbeater.cc:511] Master 127.24.141.62:41639 requested a full tablet report, sending...
I20260620 12:18:14.104207 11361 ts_manager.cc:194] Registered new tserver with Master: f8fe0f984c6d49f7b284f9d7c4ca5501 (127.24.141.3:33347)
I20260620 12:18:14.104931 11361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.3:35413
I20260620 12:18:14.108335 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:18:14.115808 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.110909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.24.141.62:41639)
0620 12:18:14.111109 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.111115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.111232 (+ 117us) server_negotiation.cc:407] Connection header received
0620 12:18:14.111420 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.111423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.111518 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.111602 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:14.111980 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.112588 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.113658 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.113904 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.114019 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.115187 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:14.115191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:14.115193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:14.115210 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:14.115237 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:14.115245 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:14.115361 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.115548 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.115609 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 12:18:14.119812 11361 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:18:14.123294 11361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:18:14.141880 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.135744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:45039 (local address 127.0.0.1:45174)
0620 12:18:14.136077 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:14.136089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:14.136173 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:14.136980 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:14.136983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:14.137001 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:14.137134 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.137142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.138758 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.138761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.139923 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.139937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.141219 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.141223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.141326 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.141361 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 12:18:14.141412 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 12:18:14.141616 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":109,"thread_start_us":111,"threads_started":1}
I20260620 12:18:14.142349 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.134742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45355 (local address 127.0.0.1:47072)
0620 12:18:14.135377 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:14.135438 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:14.135610 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:14.136517 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:14.136524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:14.136555 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:14.136744 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.136752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.138574 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.138579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.139882 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.139895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.140910 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.140914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.141160 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.141213 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 12:18:14.141349 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 12:18:14.141602 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":226,"thread_start_us":131,"threads_started":1}
I20260620 12:18:14.142819 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.134849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.24.141.1:45355)
0620 12:18:14.135934 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.135941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.135969 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:18:14.136059 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.136064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.136323 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.136920 (+ 597us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:14.137108 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.138424 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.140049 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.140759 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.140884 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.141209 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.141831 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.142369 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":68,"threads_started":1}
I20260620 12:18:14.143085 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.135050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:33347 (local address 127.0.0.1:40940)
0620 12:18:14.136233 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:14.136253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:14.136380 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:14.136855 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:14.136858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:14.136878 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:14.137016 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.137024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.138545 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.138548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.141137 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.141149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.142661 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.142665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.142757 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.142781 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:18:14.142868 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:18:14.142959 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":4,"thread_start_us":136,"threads_started":1}
I20260620 12:18:14.143215 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.135950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.24.141.2:45039)
0620 12:18:14.136504 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.136517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.136542 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:18:14.136614 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.136619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.136817 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.137281 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:14.137454 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.138631 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.140099 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.141016 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.142664 (+ 1648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.142888 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.142945 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.143061 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":96,"threads_started":1}
I20260620 12:18:14.143327 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.135773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.24.141.3:33347)
0620 12:18:14.136370 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.136383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.136412 (+ 29us) server_negotiation.cc:407] Connection header received
0620 12:18:14.136515 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.136520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.136704 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.136874 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:14.137166 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.138377 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.141327 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.142229 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.142345 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.142589 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.142976 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.143146 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":192,"threads_started":1}
I20260620 12:18:14.146782 11614 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7bcd5a8b124b0582cbe2ba501ff386 (DEFAULT_TABLE table=test-table [id=c464114d27c34839aa2bfd5c8d02c603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:14.146782 11745 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7bcd5a8b124b0582cbe2ba501ff386 (DEFAULT_TABLE table=test-table [id=c464114d27c34839aa2bfd5c8d02c603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:14.146823 11483 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7bcd5a8b124b0582cbe2ba501ff386 (DEFAULT_TABLE table=test-table [id=c464114d27c34839aa2bfd5c8d02c603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:14.148177 11483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a7bcd5a8b124b0582cbe2ba501ff386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:14.148185 11614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a7bcd5a8b124b0582cbe2ba501ff386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:14.148196 11745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a7bcd5a8b124b0582cbe2ba501ff386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:14.156560 11828 tablet_bootstrap.cc:492] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: Bootstrap starting.
I20260620 12:18:14.158046 11829 tablet_bootstrap.cc:492] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Bootstrap starting.
I20260620 12:18:14.158215 11830 tablet_bootstrap.cc:492] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Bootstrap starting.
I20260620 12:18:14.160607 11828 tablet_bootstrap.cc:654] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:14.160967 11830 tablet_bootstrap.cc:654] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:14.160967 11829 tablet_bootstrap.cc:654] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:14.161655 11828 log.cc:824] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:14.162173 11830 log.cc:824] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:14.162179 11829 log.cc:824] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:14.164311 11828 tablet_bootstrap.cc:492] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: No bootstrap required, opened a new log
I20260620 12:18:14.164575 11828 ts_tablet_manager.cc:1397] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:18:14.165011 11829 tablet_bootstrap.cc:492] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: No bootstrap required, opened a new log
I20260620 12:18:14.165418 11829 ts_tablet_manager.cc:1397] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 12:18:14.165581 11830 tablet_bootstrap.cc:492] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: No bootstrap required, opened a new log
I20260620 12:18:14.165807 11830 ts_tablet_manager.cc:1397] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 12:18:14.169096 11828 raft_consensus.cc:348] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.169414 11828 raft_consensus.cc:374] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:14.169471 11828 raft_consensus.cc:729] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bdff712cc5474e8e78deea69acf0ca, State: Initialized, Role: FOLLOWER
I20260620 12:18:14.170008 11828 consensus_queue.cc:260] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.170619 11829 raft_consensus.cc:348] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.170931 11549 heartbeater.cc:503] Master 127.24.141.62:41639 was elected leader, sending a full tablet report...
I20260620 12:18:14.171030 11829 raft_consensus.cc:374] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:14.171105 11829 raft_consensus.cc:729] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fe0f984c6d49f7b284f9d7c4ca5501, State: Initialized, Role: FOLLOWER
I20260620 12:18:14.171294 11828 ts_tablet_manager.cc:1428] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 12:18:14.171757 11829 consensus_queue.cc:260] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.170619 11830 raft_consensus.cc:348] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.172586 11830 raft_consensus.cc:374] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:14.172652 11830 raft_consensus.cc:729] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfea0e62a31746cfa3718fe4004c50c8, State: Initialized, Role: FOLLOWER
I20260620 12:18:14.172812 11811 heartbeater.cc:503] Master 127.24.141.62:41639 was elected leader, sending a full tablet report...
I20260620 12:18:14.173234 11829 ts_tablet_manager.cc:1428] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260620 12:18:14.173286 11830 consensus_queue.cc:260] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.174468 11830 ts_tablet_manager.cc:1428] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260620 12:18:14.175151 11680 heartbeater.cc:503] Master 127.24.141.62:41639 was elected leader, sending a full tablet report...
W20260620 12:18:14.336694 11812 tablet.cc:2369] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:14.339831 11834 raft_consensus.cc:482] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:18:14.340144 11834 raft_consensus.cc:504] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.341848 11834 leader_election.cc:288] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfea0e62a31746cfa3718fe4004c50c8 (127.24.141.2:45039), f8fe0f984c6d49f7b284f9d7c4ca5501 (127.24.141.3:33347)
I20260620 12:18:14.346200 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.342296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:37073 (local address 127.24.141.2:45039)
0620 12:18:14.342522 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.342528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.342833 (+ 305us) server_negotiation.cc:407] Connection header received
0620 12:18:14.343010 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.343013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.343067 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.343134 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:14.343454 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.344008 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.345206 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.345621 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.345740 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.345804 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.345965 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.346047 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 12:18:14.346228 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.342178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:45039 (local address 127.24.141.1:37073)
0620 12:18:14.342713 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:14.342729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:14.342881 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:14.343157 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:14.343160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:14.343179 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:14.343344 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.343350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.344110 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.344114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.345054 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.345094 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.345741 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.345745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.345842 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.345869 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:18:14.345935 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 12:18:14.346038 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":1,"thread_start_us":217,"threads_started":1}
I20260620 12:18:14.346619 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.342359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:33347 (local address 127.24.141.1:34003)
0620 12:18:14.342812 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:14.342829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:14.342976 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:14.343240 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:14.343242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:14.343261 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:14.343426 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.343432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.344409 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.344413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.345372 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:14.345382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.346079 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.346084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:14.346190 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.346223 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 12:18:14.346305 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 12:18:14.346419 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":185,"threads_started":1}
I20260620 12:18:14.346980 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.342399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:34003 (local address 127.24.141.3:33347)
0620 12:18:14.342561 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.342567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.342923 (+ 356us) server_negotiation.cc:407] Connection header received
0620 12:18:14.343088 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.343091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.343135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.343198 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:14.343538 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.344269 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.345504 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.345942 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.346064 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.346123 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.346719 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.346837 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:18:14.347038 11634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7bcd5a8b124b0582cbe2ba501ff386" candidate_uuid: "15bdff712cc5474e8e78deea69acf0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfea0e62a31746cfa3718fe4004c50c8" is_pre_election: true
I20260620 12:18:14.347460 11634 raft_consensus.cc:2454] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15bdff712cc5474e8e78deea69acf0ca in term 0.
I20260620 12:18:14.347667 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7bcd5a8b124b0582cbe2ba501ff386" candidate_uuid: "15bdff712cc5474e8e78deea69acf0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" is_pre_election: true
I20260620 12:18:14.347960 11765 raft_consensus.cc:2454] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15bdff712cc5474e8e78deea69acf0ca in term 0.
I20260620 12:18:14.348165 11437 leader_election.cc:302] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bdff712cc5474e8e78deea69acf0ca, dfea0e62a31746cfa3718fe4004c50c8; no voters:
I20260620 12:18:14.348577 11834 raft_consensus.cc:2781] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:18:14.348665 11834 raft_consensus.cc:482] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:18:14.348716 11834 raft_consensus.cc:3037] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:14.350044 11834 raft_consensus.cc:504] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.350474 11834 leader_election.cc:288] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [CANDIDATE]: Term 1 election: Requested vote from peers dfea0e62a31746cfa3718fe4004c50c8 (127.24.141.2:45039), f8fe0f984c6d49f7b284f9d7c4ca5501 (127.24.141.3:33347)
I20260620 12:18:14.351425 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7bcd5a8b124b0582cbe2ba501ff386" candidate_uuid: "15bdff712cc5474e8e78deea69acf0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501"
I20260620 12:18:14.351637 11765 raft_consensus.cc:3037] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:14.352838 11765 raft_consensus.cc:2454] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15bdff712cc5474e8e78deea69acf0ca in term 1.
I20260620 12:18:14.353359 11438 leader_election.cc:302] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bdff712cc5474e8e78deea69acf0ca, f8fe0f984c6d49f7b284f9d7c4ca5501; no voters:
I20260620 12:18:14.353323 11634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7bcd5a8b124b0582cbe2ba501ff386" candidate_uuid: "15bdff712cc5474e8e78deea69acf0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfea0e62a31746cfa3718fe4004c50c8"
I20260620 12:18:14.353510 11634 raft_consensus.cc:3037] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:14.353729 11834 raft_consensus.cc:2781] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:14.354089 11834 raft_consensus.cc:686] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 1 LEADER]: Becoming Leader. State: Replica: 15bdff712cc5474e8e78deea69acf0ca, State: Running, Role: LEADER
I20260620 12:18:14.354571 11834 consensus_queue.cc:237] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } }
I20260620 12:18:14.355014 11634 raft_consensus.cc:2454] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15bdff712cc5474e8e78deea69acf0ca in term 1.
I20260620 12:18:14.359141 11361 catalog_manager.cc:5495] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 15bdff712cc5474e8e78deea69acf0ca (127.24.141.1). New cstate: current_term: 1 leader_uuid: "15bdff712cc5474e8e78deea69acf0ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bdff712cc5474e8e78deea69acf0ca" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 } health_report { overall_health: UNKNOWN } } }
W20260620 12:18:14.363690 11681 tablet.cc:2369] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:14.387084 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.382636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.24.141.1:45355)
0620 12:18:14.382801 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:14.382805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:14.382869 (+ 64us) server_negotiation.cc:407] Connection header received
0620 12:18:14.382958 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:14.382960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:14.383030 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:14.383132 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:14.383446 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.384001 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.385138 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:14.385413 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:14.385534 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:14.386025 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:14.386522 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:14.386667 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:14.386881 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 12:18:14.386942 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 12:18:14.390038 11550 tablet.cc:2369] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:14.394763 11634 raft_consensus.cc:1261] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 1 FOLLOWER]: Refusing update from remote peer 15bdff712cc5474e8e78deea69acf0ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:14.394860 11765 raft_consensus.cc:1261] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 1 FOLLOWER]: Refusing update from remote peer 15bdff712cc5474e8e78deea69acf0ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:14.395718 11834 consensus_queue.cc:1035] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfea0e62a31746cfa3718fe4004c50c8" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:14.396003 11839 consensus_queue.cc:1035] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8fe0f984c6d49f7b284f9d7c4ca5501" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 33347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:14.410624 11843 mvcc.cc:205] Tried to move back new op lower bound from 7298899535431680000 to 7298899535280275456. Current Snapshot: MvccSnapshot[applied={T|T < 7298899535431680000 or (T in {7298899535431680000})}]
I20260620 12:18:14.418526 11355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:14.418728 11355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:14.422312 11355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76b3ff8090f44bc4ad144ab876e0519a: Sending DeleteTablet for 3 replicas of tablet 0a7bcd5a8b124b0582cbe2ba501ff386
I20260620 12:18:14.423496 11614 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7bcd5a8b124b0582cbe2ba501ff386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:14 UTC) from {username='slave'} at 127.0.0.1:45174
I20260620 12:18:14.423589 11745 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7bcd5a8b124b0582cbe2ba501ff386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:14 UTC) from {username='slave'} at 127.0.0.1:40940
I20260620 12:18:14.423894 11483 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7bcd5a8b124b0582cbe2ba501ff386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:14 UTC) from {username='slave'} at 127.0.0.1:47072
I20260620 12:18:14.424157 11852 tablet_replica.cc:330] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: stopping tablet replica
I20260620 12:18:14.424542 11852 raft_consensus.cc:2229] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:14.424676 11854 tablet_replica.cc:330] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca: stopping tablet replica
I20260620 12:18:14.424957 11852 raft_consensus.cc:2258] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:14.424981 11854 raft_consensus.cc:2229] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:18:14.425485 11853 tablet_replica.cc:330] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: stopping tablet replica
I20260620 12:18:14.425746 11854 raft_consensus.cc:2258] T 0a7bcd5a8b124b0582cbe2ba501ff386 P 15bdff712cc5474e8e78deea69acf0ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:14.425786 11853 raft_consensus.cc:2229] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:14.426082 11853 raft_consensus.cc:2258] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:14.426100 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11421
I20260620 12:18:14.429013 11853 ts_tablet_manager.cc:1905] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:14.429657 11852 ts_tablet_manager.cc:1905] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:14.432301 11852 ts_tablet_manager.cc:1918] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:14.432301 11853 ts_tablet_manager.cc:1918] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:14.432440 11852 log.cc:1196] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/0a7bcd5a8b124b0582cbe2ba501ff386
I20260620 12:18:14.432440 11853 log.cc:1196] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/0a7bcd5a8b124b0582cbe2ba501ff386
I20260620 12:18:14.432762 11852 ts_tablet_manager.cc:1939] T 0a7bcd5a8b124b0582cbe2ba501ff386 P dfea0e62a31746cfa3718fe4004c50c8: Deleting consensus metadata
I20260620 12:18:14.432762 11853 ts_tablet_manager.cc:1939] T 0a7bcd5a8b124b0582cbe2ba501ff386 P f8fe0f984c6d49f7b284f9d7c4ca5501: Deleting consensus metadata
W20260620 12:18:14.434671 11346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:45355 (error 108)
I20260620 12:18:14.434791 11347 catalog_manager.cc:4841] TS f8fe0f984c6d49f7b284f9d7c4ca5501 (127.24.141.3:33347): tablet 0a7bcd5a8b124b0582cbe2ba501ff386 (table test-table [id=c464114d27c34839aa2bfd5c8d02c603]) successfully deleted
I20260620 12:18:14.434927 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11552
I20260620 12:18:14.435319 11346 catalog_manager.cc:4841] TS dfea0e62a31746cfa3718fe4004c50c8 (127.24.141.2:45039): tablet 0a7bcd5a8b124b0582cbe2ba501ff386 (table test-table [id=c464114d27c34839aa2bfd5c8d02c603]) successfully deleted
I20260620 12:18:14.437278 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:14.436756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:45355 (local address 127.0.0.1:47088)
0620 12:18:14.436966 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:14.437134 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:45355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 12:18:14.437556 11346 catalog_manager.cc:4584] TS 15bdff712cc5474e8e78deea69acf0ca (127.24.141.1:45355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a7bcd5a8b124b0582cbe2ba501ff386: Network error: Client connection negotiation failed: client connection to 127.24.141.1:45355: connect: Connection refused (error 111)
I20260620 12:18:14.442435 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11683
I20260620 12:18:14.449347 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11330
2026-06-20T12:18:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1374 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:14 dist-test-slave-rh5v krb5kdc[11858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:14 dist-test-slave-rh5v krb5kdc[11858](info): set up 2 sockets
Jun 20 12:18:14 dist-test-slave-rh5v krb5kdc[11858](info): commencing operation
krb5kdc: starting...
W20260620 12:18:15.445741 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:15 dist-test-slave-rh5v krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:15.584827 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:42697
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:15.724735 11871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:15.725129 11871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:15.725186 11871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:15.729561 11871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:15.729648 11871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:15.729669 11871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:15.729686 11871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:15.729701 11871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:15.735965 11871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:15.737450 11871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:15.738756 11871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:15.748936 11876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:15.749001 11879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:15.749168 11877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:15.749212 11871 server_base.cc:1034] running on GCE node
I20260620 12:18:15.750204 11871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:15.751562 11871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:15.752760 11871 hybrid_clock.cc:648] HybridClock initialized: now 1781957895752729 us; error 50 us; skew 500 ppm
Jun 20 12:18:15 dist-test-slave-rh5v krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957895, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:15.757012 11871 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:15.758308 11871 webserver.cc:460] Webserver started at http://127.24.141.62:34689/ using document root <none> and password file <none>
I20260620 12:18:15.759003 11871 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:15.759086 11871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:15.759318 11871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:15.761864 11871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "0b0fb6ae0def4cdc80ffe0657993f926"
format_stamp: "Formatted at 2026-06-20 12:18:15 on dist-test-slave-rh5v"
server_key: "d7fc1141cd568c96faeb92daed7e1f88"
server_key_iv: "2cf6524fc5883fbcb66481228e708775"
server_key_version: "encryptionkey@0"
I20260620 12:18:15.762539 11871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "0b0fb6ae0def4cdc80ffe0657993f926"
format_stamp: "Formatted at 2026-06-20 12:18:15 on dist-test-slave-rh5v"
server_key: "d7fc1141cd568c96faeb92daed7e1f88"
server_key_iv: "2cf6524fc5883fbcb66481228e708775"
server_key_version: "encryptionkey@0"
I20260620 12:18:15.767896 11871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:15.771586 11886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:15.773797 11871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:15.774006 11871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "0b0fb6ae0def4cdc80ffe0657993f926"
format_stamp: "Formatted at 2026-06-20 12:18:15 on dist-test-slave-rh5v"
server_key: "d7fc1141cd568c96faeb92daed7e1f88"
server_key_iv: "2cf6524fc5883fbcb66481228e708775"
server_key_version: "encryptionkey@0"
I20260620 12:18:15.774155 11871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:15.796600 11871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:15.800392 11871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:15.800613 11871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:15.809442 11871 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:42697
I20260620 12:18:15.809475 11938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:42697 every 8 connection(s)
I20260620 12:18:15.810653 11871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:15.814270 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11871
I20260620 12:18:15.814460 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:15.814795 11939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:15.814823 25140 external_mini_cluster.cc:1428] Setting key fdd63b6be77ca6bcd0c1b8f0c75435a2
I20260620 12:18:15.823078 11939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926: Bootstrap starting.
Jun 20 12:18:15 dist-test-slave-rh5v krb5kdc[11858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:15.826071 11939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:15.827101 11939 log.cc:824] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:15.830394 11939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926: No bootstrap required, opened a new log
I20260620 12:18:15.832986 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:15.816755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.24.141.62:42697)
0620 12:18:15.817450 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:15.817471 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:15.817522 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:18:15.818357 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:15.818378 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:15.818803 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:15.819171 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:15.820089 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:15.821245 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:15.822231 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:15.822788 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:15.822953 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:15.825810 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:15.825817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:15.825837 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:15.825882 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:15.828440 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:15.829232 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:15.829240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:15.829247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:15.829434 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:15.829860 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:15.829865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:15.829866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:15.830464 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:15.830792 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:15.831203 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 12:18:15.831635 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":239,"threads_started":1}
I20260620 12:18:15.835676 11939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42697 } }
I20260620 12:18:15.836108 11939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:15.836167 11939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b0fb6ae0def4cdc80ffe0657993f926, State: Initialized, Role: FOLLOWER
I20260620 12:18:15.836783 11939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42697 } }
I20260620 12:18:15.836951 11939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:15.837011 11939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:15.837152 11939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:15.838505 11939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42697 } }
I20260620 12:18:15.838951 11939 leader_election.cc:302] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b0fb6ae0def4cdc80ffe0657993f926; no voters:
I20260620 12:18:15.839346 11939 leader_election.cc:288] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:15.839564 11944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:15.839962 11944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [term 1 LEADER]: Becoming Leader. State: Replica: 0b0fb6ae0def4cdc80ffe0657993f926, State: Running, Role: LEADER
I20260620 12:18:15.840392 11944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42697 } }
I20260620 12:18:15.840986 11939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:15.842391 11946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0fb6ae0def4cdc80ffe0657993f926. Latest consensus state: current_term: 1 leader_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42697 } } }
I20260620 12:18:15.842568 11946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:15.842711 11945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0fb6ae0def4cdc80ffe0657993f926" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 42697 } } }
I20260620 12:18:15.842800 11945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:15.843101 11948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:15.847123 11948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:15.855125 11948 catalog_manager.cc:1347] Generated new cluster ID: 66c7fbc975c94752ba0ccd012e406637
I20260620 12:18:15.855245 11948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:15.868866 11948 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:15.870666 11948 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:15.884706 11948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926: Generated new TSK 0
I20260620 12:18:15.886019 11948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.141.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
Entry for principal kudu/127.24.141.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab.
I20260620 12:18:15.921020 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42697
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:16.055303 11965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:16.055660 11965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:16.055738 11965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:16.060916 11965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:16.061018 11965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:16.061131 11965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:18:16.067595 11965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.1.keytab
--principal=kudu/127.24.141.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:16.068919 11965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:16.070153 11965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:16.081460 11971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:16.081517 11973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:16.081460 11970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:16.081804 11965 server_base.cc:1034] running on GCE node
I20260620 12:18:16.082479 11965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:16.083210 11965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:16.084436 11965 hybrid_clock.cc:648] HybridClock initialized: now 1781957896084394 us; error 55 us; skew 500 ppm
Jun 20 12:18:16 dist-test-slave-rh5v krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:16.087694 11965 init.cc:377] Logged in from keytab as kudu/127.24.141.1@KRBTEST.COM (short username kudu)
I20260620 12:18:16.088719 11965 webserver.cc:460] Webserver started at http://127.24.141.1:45813/ using document root <none> and password file <none>
I20260620 12:18:16.089424 11965 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:16.089500 11965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:16.089722 11965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:16.091948 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "d2d9092d96f04f6281d4b8a3117dbf96"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "09c2c38a70ab41ed0b223658855d773a"
server_key_iv: "9c9fa9e95aa47286f738b6df4903ac16"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.092549 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "d2d9092d96f04f6281d4b8a3117dbf96"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "09c2c38a70ab41ed0b223658855d773a"
server_key_iv: "9c9fa9e95aa47286f738b6df4903ac16"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.097628 11965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 12:18:16.101192 11980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.103129 11965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:16.103328 11965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "d2d9092d96f04f6281d4b8a3117dbf96"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "09c2c38a70ab41ed0b223658855d773a"
server_key_iv: "9c9fa9e95aa47286f738b6df4903ac16"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.103472 11965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:16.117787 11965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:16.121441 11965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:16.121645 11965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:16.122830 11965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:16.123910 11965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:16.123971 11965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.124034 11965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:16.124064 11965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.134289 11965 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:41141
I20260620 12:18:16.134346 12093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:41141 every 8 connection(s)
I20260620 12:18:16.135385 11965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:18:16.140467 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 11965
I20260620 12:18:16.140612 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:18:16.140941 25140 external_mini_cluster.cc:1428] Setting key 23e8e9a05a816bc721081c72af775d10
Jun 20 12:18:16 dist-test-slave-rh5v krb5kdc[11858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.1@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.141.2@KRBTEST.COM; defaulting to no policy
I20260620 12:18:16.153023 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.137905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:48485 (local address 127.24.141.62:42697)
0620 12:18:16.138098 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.138103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.139216 (+ 1113us) server_negotiation.cc:407] Connection header received
0620 12:18:16.140817 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.140822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.140906 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.140967 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:16.143013 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.143833 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.145368 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.145763 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.145867 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.148716 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:16.148719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:16.148722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:16.148751 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:16.150468 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.151046 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.151050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.151052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.151137 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.151486 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.151489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.151491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.151740 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:16.151847 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.152706 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.152862 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.2@KRBTEST.COM" created.
I20260620 12:18:16.154225 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.138351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:42697 (local address 127.24.141.1:48485)
0620 12:18:16.138980 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.139044 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.140478 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.141459 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.141467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.142088 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:16.142829 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.142848 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.144032 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.144036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.145232 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.145244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.145946 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.145951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.146148 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.146829 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:16.146848 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:16.148520 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:16.150594 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:16.150601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:16.150625 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:16.150951 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:16.151223 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:16.151226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:16.151228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:16.151399 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:16.151946 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:16.151957 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:16.152366 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.152651 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.153052 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":131,"threads_started":1}
I20260620 12:18:16.155722 12094 heartbeater.cc:347] Connected to a master server at 127.24.141.62:42697
I20260620 12:18:16.156008 12094 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:16.156592 12094 heartbeater.cc:511] Master 127.24.141.62:42697 requested a full tablet report, sending...
I20260620 12:18:16.158548 11902 ts_manager.cc:194] Registered new tserver with Master: d2d9092d96f04f6281d4b8a3117dbf96 (127.24.141.1:41141)
I20260620 12:18:16.159880 11902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.1@KRBTEST.COM'} at 127.24.141.1:48485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
Entry for principal kudu/127.24.141.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab.
I20260620 12:18:16.170504 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42697
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:16.309259 12099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:16.309661 12099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:16.309732 12099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:16.314431 12099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:16.314549 12099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:16.314647 12099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:18:16.321199 12099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.2.keytab
--principal=kudu/127.24.141.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:16.322829 12099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:16.324313 12099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:16.336593 12105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:16.336640 12107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:16.336593 12104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:16.337013 12099 server_base.cc:1034] running on GCE node
I20260620 12:18:16.337561 12099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:16.338436 12099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:16.339655 12099 hybrid_clock.cc:648] HybridClock initialized: now 1781957896339617 us; error 52 us; skew 500 ppm
Jun 20 12:18:16 dist-test-slave-rh5v krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:16.343143 12099 init.cc:377] Logged in from keytab as kudu/127.24.141.2@KRBTEST.COM (short username kudu)
I20260620 12:18:16.344405 12099 webserver.cc:460] Webserver started at http://127.24.141.2:37059/ using document root <none> and password file <none>
I20260620 12:18:16.345155 12099 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:16.345235 12099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:16.345489 12099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:16.348150 12099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "0ef334958a9448aa9c90ba0ebfdf1661"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "61617de17275bc6e8fd87e92d8df02d9"
server_key_iv: "45065fdb90a4e3c51c3cc554cd29abf2"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.348811 12099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "0ef334958a9448aa9c90ba0ebfdf1661"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "61617de17275bc6e8fd87e92d8df02d9"
server_key_iv: "45065fdb90a4e3c51c3cc554cd29abf2"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.353893 12099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:18:16.357801 12114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.359804 12099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:16.360024 12099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "0ef334958a9448aa9c90ba0ebfdf1661"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "61617de17275bc6e8fd87e92d8df02d9"
server_key_iv: "45065fdb90a4e3c51c3cc554cd29abf2"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.360189 12099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:16.377993 12099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:16.382602 12099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:16.382833 12099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:16.383646 12099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:16.384793 12099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:16.384847 12099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.384910 12099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:16.384935 12099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.396785 12099 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:38123
I20260620 12:18:16.396842 12227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:38123 every 8 connection(s)
I20260620 12:18:16.398099 12099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:18:16.400133 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 12099
I20260620 12:18:16.400274 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:18:16.400534 25140 external_mini_cluster.cc:1428] Setting key 4b4b57cb585f9644a5f254b8f2f528f3
WARNING: no policy specified for kudu/127.24.141.3@KRBTEST.COM; defaulting to no policy
Jun 20 12:18:16 dist-test-slave-rh5v krb5kdc[11858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.2@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.3@KRBTEST.COM" created.
I20260620 12:18:16.419664 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.400794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:38937 (local address 127.24.141.62:42697)
0620 12:18:16.401021 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.401027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.402539 (+ 1512us) server_negotiation.cc:407] Connection header received
0620 12:18:16.404646 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.404650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.404723 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.404815 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:16.407061 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.407779 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.410540 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.410843 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.410971 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.414712 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:16.414717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:16.414720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:16.414768 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:16.416683 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.417589 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.417592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.417594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.417697 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.418224 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.418229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.418230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.418505 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:16.418613 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.419286 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.419479 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 12:18:16.420542 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.401509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:42697 (local address 127.24.141.2:38937)
0620 12:18:16.402306 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.402399 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.404281 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.405125 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.405137 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.405768 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:16.406687 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.406716 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.408221 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.408226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.410236 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.410270 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.411098 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.411104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.411408 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.412259 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:16.412287 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:16.414338 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:16.416898 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:16.416908 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:16.416934 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:16.417444 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:16.417797 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:16.417799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:16.417801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:16.418066 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:16.418625 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:16.418634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:16.418951 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.419138 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.419529 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":156,"threads_started":1}
I20260620 12:18:16.422462 12228 heartbeater.cc:347] Connected to a master server at 127.24.141.62:42697
I20260620 12:18:16.422842 12228 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:16.423516 12228 heartbeater.cc:511] Master 127.24.141.62:42697 requested a full tablet report, sending...
I20260620 12:18:16.424965 11902 ts_manager.cc:194] Registered new tserver with Master: 0ef334958a9448aa9c90ba0ebfdf1661 (127.24.141.2:38123)
I20260620 12:18:16.425635 11902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.2@KRBTEST.COM'} at 127.24.141.2:38937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
Entry for principal kudu/127.24.141.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab.
I20260620 12:18:16.437448 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42697
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:16.574484 12233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:16.574831 12233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:16.574882 12233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:16.579452 12233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:16.579555 12233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:16.579627 12233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:18:16.585863 12233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.3.keytab
--principal=kudu/127.24.141.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:16.587396 12233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:16.588753 12233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:16.601094 12241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:16.601202 12239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:16.601172 12238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:16.601392 12233 server_base.cc:1034] running on GCE node
I20260620 12:18:16.602146 12233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:16.603078 12233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:16.604287 12233 hybrid_clock.cc:648] HybridClock initialized: now 1781957896604268 us; error 53 us; skew 500 ppm
Jun 20 12:18:16 dist-test-slave-rh5v krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:16.607935 12233 init.cc:377] Logged in from keytab as kudu/127.24.141.3@KRBTEST.COM (short username kudu)
I20260620 12:18:16.609207 12233 webserver.cc:460] Webserver started at http://127.24.141.3:41065/ using document root <none> and password file <none>
I20260620 12:18:16.609959 12233 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:16.610033 12233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:16.610252 12233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:16.612977 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "87f148aa7e59415083155d33a2ab37d9"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "0a0bb54d50c2411e2b130a3ed3a79f61"
server_key_iv: "8e89f1de58c062b863d19c33d9516db9"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.613750 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "87f148aa7e59415083155d33a2ab37d9"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "0a0bb54d50c2411e2b130a3ed3a79f61"
server_key_iv: "8e89f1de58c062b863d19c33d9516db9"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.618754 12233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:18:16.622362 12248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.624269 12233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:16.624432 12233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "87f148aa7e59415083155d33a2ab37d9"
format_stamp: "Formatted at 2026-06-20 12:18:16 on dist-test-slave-rh5v"
server_key: "0a0bb54d50c2411e2b130a3ed3a79f61"
server_key_iv: "8e89f1de58c062b863d19c33d9516db9"
server_key_version: "encryptionkey@0"
I20260620 12:18:16.624568 12233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:16.638998 12233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:16.642951 12233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:16.643157 12233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:16.643934 12233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:16.645035 12233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:16.645133 12233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.645184 12233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:16.645207 12233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:16.657647 12233 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:36865
I20260620 12:18:16.657747 12361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:36865 every 8 connection(s)
I20260620 12:18:16.658967 12233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:18:16.667362 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 12233
I20260620 12:18:16.667542 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:18:16.667877 25140 external_mini_cluster.cc:1428] Setting key 20219f677ae86b3401392014f98db54b
Jun 20 12:18:16 dist-test-slave-rh5v krb5kdc[11858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.3@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:16.677021 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.661458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:53831 (local address 127.24.141.62:42697)
0620 12:18:16.661601 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.661605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.662979 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 12:18:16.664553 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.664557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.664631 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.664711 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:16.666293 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.666871 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.668235 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.668538 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.668644 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.672139 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:16.672144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:16.672146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:16.672183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:16.674161 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.674995 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.674998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.675000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.675122 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.675594 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.675597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.675599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.675845 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:16.675938 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.676703 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.676873 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:18:16.678318 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.662016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:42697 (local address 127.24.141.3:53831)
0620 12:18:16.662737 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.662804 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.664260 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.664956 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.664965 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.665499 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:16.666096 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.666113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.667033 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.667036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.668088 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.668103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.668716 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.668722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.668947 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.669671 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:16.669697 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:16.671837 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:16.674366 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:16.674374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:16.674398 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:16.674828 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:16.675262 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:16.675265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:16.675266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:16.675470 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:16.675963 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:16.675970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:16.676346 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.676604 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.677005 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":157,"threads_started":1}
I20260620 12:18:16.680002 12362 heartbeater.cc:347] Connected to a master server at 127.24.141.62:42697
I20260620 12:18:16.680368 12362 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:16.681030 12362 heartbeater.cc:511] Master 127.24.141.62:42697 requested a full tablet report, sending...
I20260620 12:18:16.682578 11902 ts_manager.cc:194] Registered new tserver with Master: 87f148aa7e59415083155d33a2ab37d9 (127.24.141.3:36865)
I20260620 12:18:16.683348 11902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.141.3@KRBTEST.COM'} at 127.24.141.3:53831
I20260620 12:18:16.683909 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:18:16.694555 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.686549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.24.141.62:42697)
0620 12:18:16.686734 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.686738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.686870 (+ 132us) server_negotiation.cc:407] Connection header received
0620 12:18:16.687036 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.687040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.687088 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.687156 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:16.687933 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.688463 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.689367 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.689619 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.689713 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.691184 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:16.691187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:16.691189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:16.691218 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:16.692457 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.692911 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.692914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.692915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.692995 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:16.693296 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:16.693300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:16.693301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:16.693573 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:16.693699 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.694118 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.694339 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:18:16.698800 11902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:18:16.702458 11902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:18:16.722474 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.715790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.24.141.1:41141)
0620 12:18:16.716387 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.716392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.716410 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:18:16.716495 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.716500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.716717 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.716894 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:16.717373 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.718493 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.720640 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.721504 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.721617 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.721751 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.722186 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.722329 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":146,"threads_started":1}
I20260620 12:18:16.722438 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.714918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:41141 (local address 127.0.0.1:60658)
0620 12:18:16.715780 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.715856 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.716012 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.716971 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.716974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.716992 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:16.717218 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.717229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.718606 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.718610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.720457 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.720469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.721823 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.721827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.721917 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.721934 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.722024 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.722234 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":370,"thread_start_us":118,"threads_started":1}
I20260620 12:18:16.722486 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.714794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.24.141.3:36865)
0620 12:18:16.715559 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.715575 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.715965 (+ 390us) server_negotiation.cc:407] Connection header received
0620 12:18:16.716302 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.716312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.716561 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.716753 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:16.717179 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.718447 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.720366 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.721261 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.721399 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.721589 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.722112 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.722286 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":288,"threads_started":1}
I20260620 12:18:16.723034 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.715430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:38123 (local address 127.0.0.1:39890)
0620 12:18:16.716188 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.716203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.716282 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.717084 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.717087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.717104 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:16.717278 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.717284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.718649 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.718652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.720457 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.720469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.722557 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.722561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.722684 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.722712 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.722789 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.722889 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":305,"thread_start_us":105,"threads_started":1}
I20260620 12:18:16.723491 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.715856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.24.141.2:38123)
0620 12:18:16.716350 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.716362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.716387 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:18:16.716454 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.716458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.716591 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.716746 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:16.717411 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.718536 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.721199 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.722140 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.722255 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.722468 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.723228 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.723355 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":140,"threads_started":1}
I20260620 12:18:16.725378 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.714564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36865 (local address 127.0.0.1:42402)
0620 12:18:16.715779 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.715856 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.716071 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.716758 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.716766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.716800 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:16.717002 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.717012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.718636 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.718640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.720151 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.720169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.721682 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.721686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.721839 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.721889 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.722024 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.725233 (+ 3209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":3123,"thread_start_us":105,"threads_started":1}
I20260620 12:18:16.726675 12028 tablet_service.cc:1467] Processing CreateTablet for tablet b85e096b592241f5acc8d45f5a38704f (DEFAULT_TABLE table=test-table [id=6479cc53ca384432980dd523ff10be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:16.726675 12162 tablet_service.cc:1467] Processing CreateTablet for tablet b85e096b592241f5acc8d45f5a38704f (DEFAULT_TABLE table=test-table [id=6479cc53ca384432980dd523ff10be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:16.727972 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85e096b592241f5acc8d45f5a38704f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:16.727972 12162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85e096b592241f5acc8d45f5a38704f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:16.733973 12296 tablet_service.cc:1467] Processing CreateTablet for tablet b85e096b592241f5acc8d45f5a38704f (DEFAULT_TABLE table=test-table [id=6479cc53ca384432980dd523ff10be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:16.735321 12296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85e096b592241f5acc8d45f5a38704f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:16.738041 12379 tablet_bootstrap.cc:492] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96: Bootstrap starting.
I20260620 12:18:16.738159 12380 tablet_bootstrap.cc:492] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Bootstrap starting.
I20260620 12:18:16.740340 12379 tablet_bootstrap.cc:654] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:16.740901 12380 tablet_bootstrap.cc:654] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:16.741340 12379 log.cc:824] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:16.742424 12380 log.cc:824] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:16.743731 12379 tablet_bootstrap.cc:492] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96: No bootstrap required, opened a new log
I20260620 12:18:16.743966 12379 ts_tablet_manager.cc:1397] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 12:18:16.745609 12380 tablet_bootstrap.cc:492] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: No bootstrap required, opened a new log
I20260620 12:18:16.745836 12380 ts_tablet_manager.cc:1397] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 12:18:16.748096 12379 raft_consensus.cc:348] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.748399 12379 raft_consensus.cc:374] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:16.748450 12379 raft_consensus.cc:729] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d9092d96f04f6281d4b8a3117dbf96, State: Initialized, Role: FOLLOWER
I20260620 12:18:16.748932 12379 consensus_queue.cc:260] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.749150 12383 tablet_bootstrap.cc:492] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Bootstrap starting.
I20260620 12:18:16.749442 12380 raft_consensus.cc:348] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.749778 12380 raft_consensus.cc:374] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:16.749840 12380 raft_consensus.cc:729] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ef334958a9448aa9c90ba0ebfdf1661, State: Initialized, Role: FOLLOWER
I20260620 12:18:16.750064 12379 ts_tablet_manager.cc:1428] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:18:16.750509 12094 heartbeater.cc:503] Master 127.24.141.62:42697 was elected leader, sending a full tablet report...
I20260620 12:18:16.750443 12380 consensus_queue.cc:260] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.751245 12228 heartbeater.cc:503] Master 127.24.141.62:42697 was elected leader, sending a full tablet report...
I20260620 12:18:16.751694 12380 ts_tablet_manager.cc:1428] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:18:16.752147 12383 tablet_bootstrap.cc:654] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:16.753726 12383 log.cc:824] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:16.756207 12383 tablet_bootstrap.cc:492] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: No bootstrap required, opened a new log
I20260620 12:18:16.756469 12383 ts_tablet_manager.cc:1397] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:18:16.759989 12383 raft_consensus.cc:348] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.760295 12383 raft_consensus.cc:374] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:16.760346 12383 raft_consensus.cc:729] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f148aa7e59415083155d33a2ab37d9, State: Initialized, Role: FOLLOWER
I20260620 12:18:16.760906 12383 consensus_queue.cc:260] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.761854 12362 heartbeater.cc:503] Master 127.24.141.62:42697 was elected leader, sending a full tablet report...
I20260620 12:18:16.762081 12383 ts_tablet_manager.cc:1428] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:18:16.771981 12385 raft_consensus.cc:482] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:18:16.772200 12385 raft_consensus.cc:504] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.773612 12385 leader_election.cc:288] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87f148aa7e59415083155d33a2ab37d9 (127.24.141.3:36865), d2d9092d96f04f6281d4b8a3117dbf96 (127.24.141.1:41141)
I20260620 12:18:16.778162 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.774071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:36865 (local address 127.24.141.2:60265)
0620 12:18:16.774247 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.774266 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.774400 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.774728 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.774732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.774755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:16.774889 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.774896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.775669 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.775673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.776562 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.776573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.777656 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.777660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.777804 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.777832 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.777910 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.778010 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 12:18:16.778260 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.774233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:60265 (local address 127.24.141.3:36865)
0620 12:18:16.774390 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.774395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.774417 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:18:16.774539 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.774543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.774611 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.774700 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:16.775000 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.775559 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.776721 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.777391 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.777524 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.777587 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.777959 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.778089 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:18:16.778853 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.774136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:51687 (local address 127.24.141.1:41141)
0620 12:18:16.774261 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.774266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.774804 (+ 538us) server_negotiation.cc:407] Connection header received
0620 12:18:16.774945 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.774948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.774994 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.775059 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:16.775355 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.775917 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.777313 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.778009 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.778144 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.778204 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.778592 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.778723 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:18:16.778887 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.774071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:41141 (local address 127.24.141.2:51687)
0620 12:18:16.774702 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.774714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:16.774821 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:16.775082 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:16.775085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:16.775104 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:16.775252 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.775258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.776033 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.776036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.777153 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:16.777164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.778137 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.778141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:16.778231 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.778418 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 12:18:16.778536 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 12:18:16.778622 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":211,"thread_start_us":163,"threads_started":1}
I20260620 12:18:16.779287 12316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85e096b592241f5acc8d45f5a38704f" candidate_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f148aa7e59415083155d33a2ab37d9" is_pre_election: true
I20260620 12:18:16.779695 12316 raft_consensus.cc:2454] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ef334958a9448aa9c90ba0ebfdf1661 in term 0.
I20260620 12:18:16.780480 12116 leader_election.cc:302] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ef334958a9448aa9c90ba0ebfdf1661, 87f148aa7e59415083155d33a2ab37d9; no voters:
I20260620 12:18:16.780870 12385 raft_consensus.cc:2781] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:18:16.781038 12385 raft_consensus.cc:482] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:18:16.781136 12385 raft_consensus.cc:3037] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:16.780921 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85e096b592241f5acc8d45f5a38704f" candidate_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" is_pre_election: true
I20260620 12:18:16.781438 12048 raft_consensus.cc:2454] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ef334958a9448aa9c90ba0ebfdf1661 in term 0.
I20260620 12:18:16.782797 12385 raft_consensus.cc:504] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.783231 12385 leader_election.cc:288] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [CANDIDATE]: Term 1 election: Requested vote from peers 87f148aa7e59415083155d33a2ab37d9 (127.24.141.3:36865), d2d9092d96f04f6281d4b8a3117dbf96 (127.24.141.1:41141)
I20260620 12:18:16.783767 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85e096b592241f5acc8d45f5a38704f" candidate_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d9092d96f04f6281d4b8a3117dbf96"
I20260620 12:18:16.783932 12048 raft_consensus.cc:3037] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:16.784113 12316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85e096b592241f5acc8d45f5a38704f" candidate_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f148aa7e59415083155d33a2ab37d9"
I20260620 12:18:16.784340 12316 raft_consensus.cc:3037] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:16.785517 12048 raft_consensus.cc:2454] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ef334958a9448aa9c90ba0ebfdf1661 in term 1.
I20260620 12:18:16.786034 12117 leader_election.cc:302] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ef334958a9448aa9c90ba0ebfdf1661, d2d9092d96f04f6281d4b8a3117dbf96; no voters:
I20260620 12:18:16.786101 12316 raft_consensus.cc:2454] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ef334958a9448aa9c90ba0ebfdf1661 in term 1.
I20260620 12:18:16.786304 12385 raft_consensus.cc:2781] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:16.786666 12385 raft_consensus.cc:686] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 1 LEADER]: Becoming Leader. State: Replica: 0ef334958a9448aa9c90ba0ebfdf1661, State: Running, Role: LEADER
I20260620 12:18:16.787336 12385 consensus_queue.cc:237] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } }
I20260620 12:18:16.791491 11902 catalog_manager.cc:5495] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ef334958a9448aa9c90ba0ebfdf1661 (127.24.141.2). New cstate: current_term: 1 leader_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ef334958a9448aa9c90ba0ebfdf1661" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 } health_report { overall_health: UNKNOWN } } }
I20260620 12:18:16.825853 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.821509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.24.141.2:38123)
0620 12:18:16.821712 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:16.821718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:16.821741 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:18:16.821860 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:16.821864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:16.821949 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:16.822092 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:16.822325 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.823202 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.824181 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:16.824462 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:16.824576 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:16.824876 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:16.825439 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:16.825551 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:16.825616 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 12:18:16.825683 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:18:16.833957 12316 raft_consensus.cc:1261] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 1 FOLLOWER]: Refusing update from remote peer 0ef334958a9448aa9c90ba0ebfdf1661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:16.833997 12048 raft_consensus.cc:1261] T b85e096b592241f5acc8d45f5a38704f P d2d9092d96f04f6281d4b8a3117dbf96 [term 1 FOLLOWER]: Refusing update from remote peer 0ef334958a9448aa9c90ba0ebfdf1661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:16.834952 12385 consensus_queue.cc:1035] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f148aa7e59415083155d33a2ab37d9" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:16.835228 12389 consensus_queue.cc:1035] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d9092d96f04f6281d4b8a3117dbf96" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:16.846227 12394 mvcc.cc:205] Tried to move back new op lower bound from 7298899545420996608 to 7298899545244762112. Current Snapshot: MvccSnapshot[applied={T|T < 7298899545420996608}]
I20260620 12:18:16.846848 12393 mvcc.cc:205] Tried to move back new op lower bound from 7298899545420996608 to 7298899545244762112. Current Snapshot: MvccSnapshot[applied={T|T < 7298899545420996608}]
I20260620 12:18:16.858896 11903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:16.859220 11903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:16.863219 11903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b0fb6ae0def4cdc80ffe0657993f926: Sending DeleteTablet for 3 replicas of tablet b85e096b592241f5acc8d45f5a38704f
I20260620 12:18:16.864672 12296 tablet_service.cc:1514] Processing DeleteTablet for tablet b85e096b592241f5acc8d45f5a38704f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:16 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:42402
I20260620 12:18:16.864816 12028 tablet_service.cc:1514] Processing DeleteTablet for tablet b85e096b592241f5acc8d45f5a38704f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:16 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:60658
I20260620 12:18:16.864955 12162 tablet_service.cc:1514] Processing DeleteTablet for tablet b85e096b592241f5acc8d45f5a38704f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:16 UTC) from {username='kudu', principal='kudu/127.24.141.62@KRBTEST.COM'} at 127.0.0.1:39890
I20260620 12:18:16.865495 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11965
I20260620 12:18:16.865499 12403 tablet_replica.cc:330] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: stopping tablet replica
I20260620 12:18:16.866209 12403 raft_consensus.cc:2229] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:18:16.866782 12403 raft_consensus.cc:2258] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:16.868505 12402 tablet_replica.cc:330] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: stopping tablet replica
I20260620 12:18:16.868726 12403 ts_tablet_manager.cc:1905] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:16.869017 12402 raft_consensus.cc:2229] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:16.869482 12402 raft_consensus.cc:2258] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:16.871547 12402 ts_tablet_manager.cc:1905] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:16.872484 12403 ts_tablet_manager.cc:1918] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:16.872627 12403 log.cc:1196] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/b85e096b592241f5acc8d45f5a38704f
I20260620 12:18:16.872972 12403 ts_tablet_manager.cc:1939] T b85e096b592241f5acc8d45f5a38704f P 0ef334958a9448aa9c90ba0ebfdf1661: Deleting consensus metadata
I20260620 12:18:16.875173 11888 catalog_manager.cc:4841] TS 0ef334958a9448aa9c90ba0ebfdf1661 (127.24.141.2:38123): tablet b85e096b592241f5acc8d45f5a38704f (table test-table [id=6479cc53ca384432980dd523ff10be11]) successfully deleted
I20260620 12:18:16.875993 12402 ts_tablet_manager.cc:1918] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:16.876116 12402 log.cc:1196] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/b85e096b592241f5acc8d45f5a38704f
I20260620 12:18:16.876441 12402 ts_tablet_manager.cc:1939] T b85e096b592241f5acc8d45f5a38704f P 87f148aa7e59415083155d33a2ab37d9: Deleting consensus metadata
W20260620 12:18:16.877369 11889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:41141 (error 108)
I20260620 12:18:16.877938 11888 catalog_manager.cc:4841] TS 87f148aa7e59415083155d33a2ab37d9 (127.24.141.3:36865): tablet b85e096b592241f5acc8d45f5a38704f (table test-table [id=6479cc53ca384432980dd523ff10be11]) successfully deleted
I20260620 12:18:16.878320 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 12099
I20260620 12:18:16.880775 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:16.880446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:41141 (local address 127.0.0.1:60670)
0620 12:18:16.880583 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:16.880676 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 12:18:16.880971 11889 catalog_manager.cc:4584] TS d2d9092d96f04f6281d4b8a3117dbf96 (127.24.141.1:41141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b85e096b592241f5acc8d45f5a38704f: Network error: Client connection negotiation failed: client connection to 127.24.141.1:41141: connect: Connection refused (error 111)
I20260620 12:18:16.886318 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 12233
I20260620 12:18:16.893344 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 11871
2026-06-20T12:18:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2444 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T12:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:16Z Disabled control of system clock
I20260620 12:18:16.928999 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33393
--webserver_interface=127.24.141.62
--webserver_port=0
--builtin_ntp_servers=127.24.141.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:33393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:18:17.072113 12409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:17.072487 12409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:17.072528 12409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:17.076924 12409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:17.077028 12409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:17.077044 12409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:17.077081 12409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:17.077098 12409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:17.082530 12409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:33393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:17.084093 12409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:17.086062 12409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:17.096940 12415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:17.096949 12417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:17.096949 12414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:17.097554 12409 server_base.cc:1034] running on GCE node
I20260620 12:18:17.098356 12409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:17.099884 12409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:17.101106 12409 hybrid_clock.cc:648] HybridClock initialized: now 1781957897101064 us; error 68 us; skew 500 ppm
I20260620 12:18:17.103669 12409 webserver.cc:460] Webserver started at http://127.24.141.62:45485/ using document root <none> and password file <none>
I20260620 12:18:17.104441 12409 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:17.104522 12409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:17.104772 12409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:17.107537 12409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "acb6a53aba044ed8884319728577bcc3"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "e34889f4074eb4e08fdd8fcd91ebed10"
server_key_iv: "2ff1e8e0c065da8489a2aea1dbbe9edc"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.108234 12409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "acb6a53aba044ed8884319728577bcc3"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "e34889f4074eb4e08fdd8fcd91ebed10"
server_key_iv: "2ff1e8e0c065da8489a2aea1dbbe9edc"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.113250 12409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:18:17.116664 12423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.118414 12409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:18:17.118569 12409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "acb6a53aba044ed8884319728577bcc3"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "e34889f4074eb4e08fdd8fcd91ebed10"
server_key_iv: "2ff1e8e0c065da8489a2aea1dbbe9edc"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.118718 12409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:17.151589 12409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:17.152587 12409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:17.152796 12409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:17.161772 12409 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:33393
I20260620 12:18:17.161790 12475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:33393 every 8 connection(s)
I20260620 12:18:17.163106 12409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:17.167029 12476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:17.169497 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 12409
I20260620 12:18:17.169641 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:17.169972 25140 external_mini_cluster.cc:1428] Setting key c962a3de2d649ecaa5f7a5e7bbc1c73a
I20260620 12:18:17.174818 12476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3: Bootstrap starting.
I20260620 12:18:17.177814 12476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:17.178774 12476 log.cc:824] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:17.182029 12476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3: No bootstrap required, opened a new log
I20260620 12:18:17.182739 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.171917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.24.141.62:33393)
0620 12:18:17.172851 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.172872 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.172927 (+ 55us) server_negotiation.cc:407] Connection header received
0620 12:18:17.173944 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.173980 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.174588 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.175037 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:17.175522 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.176637 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.178059 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.178670 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.178824 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.179968 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:17.179975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:17.179999 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:17.180683 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:17.180780 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:17.180815 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:17.180939 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:17.181165 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 12:18:17.181542 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":174,"threads_started":1}
I20260620 12:18:17.186887 12476 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6a53aba044ed8884319728577bcc3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33393 } }
I20260620 12:18:17.187188 12476 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:17.187225 12476 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb6a53aba044ed8884319728577bcc3, State: Initialized, Role: FOLLOWER
I20260620 12:18:17.187681 12476 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6a53aba044ed8884319728577bcc3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33393 } }
I20260620 12:18:17.187815 12476 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:17.187860 12476 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:17.187950 12476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:17.189172 12476 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6a53aba044ed8884319728577bcc3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33393 } }
I20260620 12:18:17.189587 12476 leader_election.cc:302] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb6a53aba044ed8884319728577bcc3; no voters:
I20260620 12:18:17.189965 12476 leader_election.cc:288] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:17.190236 12481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:17.190589 12481 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [term 1 LEADER]: Becoming Leader. State: Replica: acb6a53aba044ed8884319728577bcc3, State: Running, Role: LEADER
I20260620 12:18:17.191006 12481 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6a53aba044ed8884319728577bcc3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33393 } }
I20260620 12:18:17.191531 12476 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:17.193045 12483 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb6a53aba044ed8884319728577bcc3. Latest consensus state: current_term: 1 leader_uuid: "acb6a53aba044ed8884319728577bcc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6a53aba044ed8884319728577bcc3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33393 } } }
I20260620 12:18:17.193269 12483 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:17.193564 12482 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb6a53aba044ed8884319728577bcc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6a53aba044ed8884319728577bcc3" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 33393 } } }
I20260620 12:18:17.193668 12482 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:17.194005 12491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:17.197952 12491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:17.206180 12491 catalog_manager.cc:1347] Generated new cluster ID: 33b716e721d747d1b859ee5fe03f4eec
I20260620 12:18:17.206306 12491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:17.228495 12491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:17.229707 12491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:17.234773 12491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3: Generated new TSK 0
I20260620 12:18:17.235858 12491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:17.247706 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.1:0
--local_ip_for_outbound_sockets=127.24.141.1
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33393
--builtin_ntp_servers=127.24.141.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:18:17.402230 12500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:17.402581 12500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:17.402639 12500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:17.407277 12500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:17.407368 12500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:17.407450 12500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.1
I20260620 12:18:17.412813 12500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.141.1
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.1
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:17.414228 12500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:17.415853 12500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:17.427489 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:17.427589 12506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:17.427861 12500 server_base.cc:1034] running on GCE node
W20260620 12:18:17.427562 12505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:17.428429 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:17.429345 12500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:17.430585 12500 hybrid_clock.cc:648] HybridClock initialized: now 1781957897430563 us; error 64 us; skew 500 ppm
I20260620 12:18:17.433552 12500 webserver.cc:460] Webserver started at http://127.24.141.1:34311/ using document root <none> and password file <none>
I20260620 12:18:17.434423 12500 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:17.434705 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:17.435088 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:17.437862 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data/instance:
uuid: "cfbb5ce7a710413d8ba70684952b4bc7"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "00bbe87d1b6a5fe41bac1bf26bfbeb28"
server_key_iv: "7a38441c4bc6b708198b8dfe868a2847"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.438536 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance:
uuid: "cfbb5ce7a710413d8ba70684952b4bc7"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "00bbe87d1b6a5fe41bac1bf26bfbeb28"
server_key_iv: "7a38441c4bc6b708198b8dfe868a2847"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.444119 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:17.448051 12514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.450001 12500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:17.450212 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/wal
uuid: "cfbb5ce7a710413d8ba70684952b4bc7"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "00bbe87d1b6a5fe41bac1bf26bfbeb28"
server_key_iv: "7a38441c4bc6b708198b8dfe868a2847"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.450359 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:17.489212 12500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:17.490041 12500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:17.490247 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:17.490921 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:17.492089 12500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:17.492144 12500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.492204 12500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:17.492244 12500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.503509 12500 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.1:43675
I20260620 12:18:17.503667 12627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.1:43675 every 8 connection(s)
I20260620 12:18:17.504761 12500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/data/info.pb
I20260620 12:18:17.508709 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 12500
I20260620 12:18:17.508877 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-0/wal/instance
I20260620 12:18:17.509294 25140 external_mini_cluster.cc:1428] Setting key 2a91c257314075ce318631d841d1c102
I20260620 12:18:17.512241 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.2:0
--local_ip_for_outbound_sockets=127.24.141.2
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33393
--builtin_ntp_servers=127.24.141.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:18:17.517910 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.507606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.1:57339 (local address 127.24.141.62:33393)
0620 12:18:17.507866 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.507889 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.509729 (+ 1840us) server_negotiation.cc:407] Connection header received
0620 12:18:17.511614 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.511617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.511673 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.511778 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:17.513458 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.514113 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.515542 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.515806 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.515915 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.517306 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:17.517310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:17.517312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:17.517330 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:17.517357 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:17.517367 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:17.517432 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:17.517615 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 12:18:17.517707 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:18:17.519083 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.508227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33393 (local address 127.24.141.1:57339)
0620 12:18:17.509439 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:17.509516 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:17.511288 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:17.512079 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:17.512092 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:17.512241 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:17.513219 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.513242 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.514317 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.514319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:17.515385 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.515397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.515999 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.516003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:17.516200 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.516923 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:17.516924 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:17.516981 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:17.517121 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:17.517445 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:17.517456 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:17.517463 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:18:17.517598 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 12:18:17.517988 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":201,"threads_started":1}
I20260620 12:18:17.520745 12628 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33393
I20260620 12:18:17.521130 12628 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:17.521823 12628 heartbeater.cc:511] Master 127.24.141.62:33393 requested a full tablet report, sending...
I20260620 12:18:17.523973 12440 ts_manager.cc:194] Registered new tserver with Master: cfbb5ce7a710413d8ba70684952b4bc7 (127.24.141.1:43675)
I20260620 12:18:17.525416 12440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.1:57339
W20260620 12:18:17.638110 12631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:17.638417 12631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:17.638475 12631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:17.643306 12631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:17.643388 12631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:17.643471 12631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.2
I20260620 12:18:17.648782 12631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.141.2
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.2
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:17.650050 12631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:17.651571 12631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:17.663270 12636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:17.663264 12639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:17.663257 12637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:17.663897 12631 server_base.cc:1034] running on GCE node
I20260620 12:18:17.664489 12631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:17.665454 12631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:17.666678 12631 hybrid_clock.cc:648] HybridClock initialized: now 1781957897666636 us; error 69 us; skew 500 ppm
I20260620 12:18:17.669646 12631 webserver.cc:460] Webserver started at http://127.24.141.2:40631/ using document root <none> and password file <none>
I20260620 12:18:17.670507 12631 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:17.670595 12631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:17.670845 12631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:17.673867 12631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data/instance:
uuid: "2ef7be240631431eaca6dabafeb15828"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "a9da60ae9f08c62a7d994788256de147"
server_key_iv: "c92ffee52c0ce8491fe1ba943dccec2c"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.674628 12631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance:
uuid: "2ef7be240631431eaca6dabafeb15828"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "a9da60ae9f08c62a7d994788256de147"
server_key_iv: "c92ffee52c0ce8491fe1ba943dccec2c"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.680064 12631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:17.683910 12645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.685918 12631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:17.686117 12631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal
uuid: "2ef7be240631431eaca6dabafeb15828"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "a9da60ae9f08c62a7d994788256de147"
server_key_iv: "c92ffee52c0ce8491fe1ba943dccec2c"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.686277 12631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:17.710191 12631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:17.711200 12631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:17.711406 12631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:17.712174 12631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:17.713506 12631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:17.713574 12631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.713655 12631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:17.713681 12631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.724398 12631 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.2:39531
I20260620 12:18:17.724432 12758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.2:39531 every 8 connection(s)
I20260620 12:18:17.725623 12631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/data/info.pb
I20260620 12:18:17.731395 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 12631
I20260620 12:18:17.731567 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal/instance
I20260620 12:18:17.731889 25140 external_mini_cluster.cc:1428] Setting key 83f04a84b522ec0057b36da20f47cb6d
I20260620 12:18:17.735078 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.141.3:0
--local_ip_for_outbound_sockets=127.24.141.3
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33393
--builtin_ntp_servers=127.24.141.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:18:17.738462 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.728133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:37537 (local address 127.24.141.62:33393)
0620 12:18:17.728359 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.728363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.730058 (+ 1695us) server_negotiation.cc:407] Connection header received
0620 12:18:17.731789 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.731794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.731866 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.731949 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:17.733394 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.734112 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.735637 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.735875 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.735971 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.737748 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:17.737753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:17.737757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:17.737779 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:17.737812 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:17.737824 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:17.737903 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:17.738145 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 12:18:17.738254 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 12:18:17.739737 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.728647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33393 (local address 127.24.141.2:37537)
0620 12:18:17.729831 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:17.729907 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:17.731392 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:17.732205 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:17.732215 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:17.732346 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:17.733204 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.733237 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.734267 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.734270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:17.735491 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.735508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.736052 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.736057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:17.736248 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.737258 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:17.737261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:17.737336 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:17.737466 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:17.737941 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:17.737957 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:17.737965 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:18:17.738125 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 12:18:17.738548 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":169,"threads_started":1}
I20260620 12:18:17.741582 12759 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33393
I20260620 12:18:17.742019 12759 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:17.742938 12759 heartbeater.cc:511] Master 127.24.141.62:33393 requested a full tablet report, sending...
I20260620 12:18:17.744716 12440 ts_manager.cc:194] Registered new tserver with Master: 2ef7be240631431eaca6dabafeb15828 (127.24.141.2:39531)
I20260620 12:18:17.745535 12440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.2:37537
W20260620 12:18:17.861533 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:17.861887 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:17.861948 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:17.866583 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:17.866693 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:17.866786 12762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.141.3
I20260620 12:18:17.873531 12762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.141.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.141.3
--webserver_port=0
--tserver_master_addrs=127.24.141.62:33393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.141.3
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:17.875380 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:17.877034 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:17.888430 12770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:17.888543 12762 server_base.cc:1034] running on GCE node
W20260620 12:18:17.888470 12768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:17.888451 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:17.889674 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:17.890552 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:17.891887 12762 hybrid_clock.cc:648] HybridClock initialized: now 1781957897891785 us; error 56 us; skew 500 ppm
I20260620 12:18:17.894645 12762 webserver.cc:460] Webserver started at http://127.24.141.3:35415/ using document root <none> and password file <none>
I20260620 12:18:17.895376 12762 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:17.895462 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:17.895682 12762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:17.898126 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data/instance:
uuid: "0d9ff04d558c4b1fa3983e863806abf4"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "9af68441e0e8e9781276ead4fafd6e1b"
server_key_iv: "df8205795f7a19d057e6ba8e319b5a3c"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.898787 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance:
uuid: "0d9ff04d558c4b1fa3983e863806abf4"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "9af68441e0e8e9781276ead4fafd6e1b"
server_key_iv: "df8205795f7a19d057e6ba8e319b5a3c"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.903620 12762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 12:18:17.907100 12776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.909050 12762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 12:18:17.909314 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal
uuid: "0d9ff04d558c4b1fa3983e863806abf4"
format_stamp: "Formatted at 2026-06-20 12:18:17 on dist-test-slave-rh5v"
server_key: "9af68441e0e8e9781276ead4fafd6e1b"
server_key_iv: "df8205795f7a19d057e6ba8e319b5a3c"
server_key_version: "encryptionkey@0"
I20260620 12:18:17.909461 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:17.933739 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:17.934569 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:17.934767 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:17.935425 12762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:18:17.936530 12762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:18:17.936582 12762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.936642 12762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:18:17.936668 12762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:17.946759 12762 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.3:34077
I20260620 12:18:17.946826 12889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.3:34077 every 8 connection(s)
I20260620 12:18:17.947948 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/data/info.pb
I20260620 12:18:17.955961 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 12762
I20260620 12:18:17.956154 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal/instance
I20260620 12:18:17.956483 25140 external_mini_cluster.cc:1428] Setting key b0dcae6bcac2c352385cc0fed0d74431
I20260620 12:18:17.960666 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.950725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.3:60319 (local address 127.24.141.62:33393)
0620 12:18:17.950940 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.950945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.952478 (+ 1533us) server_negotiation.cc:407] Connection header received
0620 12:18:17.954381 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.954385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.954461 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.954583 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:17.956300 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.956937 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.958461 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.958700 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.958808 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.960135 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:17.960138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:17.960140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:17.960156 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:17.960177 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:17.960186 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:17.960248 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:17.960415 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 12:18:17.960492 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 12:18:17.961688 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.951166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:33393 (local address 127.24.141.3:60319)
0620 12:18:17.952207 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:17.952276 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:17.954031 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:17.954873 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:17.954883 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:17.955089 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:17.955941 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.955967 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.957179 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.957184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:17.958276 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.958302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.958877 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.958882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:17.959078 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.959799 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:18:17.959801 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:17.959852 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:18:17.959941 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:17.960260 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:17.960270 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:17.960277 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:18:17.960408 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 12:18:17.960756 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":154,"threads_started":1}
I20260620 12:18:17.963307 12890 heartbeater.cc:347] Connected to a master server at 127.24.141.62:33393
I20260620 12:18:17.963618 12890 heartbeater.cc:464] Registering TS with master...
I20260620 12:18:17.964253 12890 heartbeater.cc:511] Master 127.24.141.62:33393 requested a full tablet report, sending...
I20260620 12:18:17.965703 12440 ts_manager.cc:194] Registered new tserver with Master: 0d9ff04d558c4b1fa3983e863806abf4 (127.24.141.3:34077)
I20260620 12:18:17.966327 12440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.141.3:60319
I20260620 12:18:17.971285 25140 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:18:17.978569 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.973977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.24.141.62:33393)
0620 12:18:17.974184 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.974189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.974299 (+ 110us) server_negotiation.cc:407] Connection header received
0620 12:18:17.974489 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.974492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.974690 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.974851 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:17.975293 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.975828 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.976779 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:17.976998 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:17.977140 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:17.978123 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:17.978126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:17.978129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:17.978144 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:17.978165 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:17.978173 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:17.978236 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:17.978330 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 12:18:17.978405 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 12:18:17.982167 12440 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:18:17.985805 12440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:18:18.004586 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.998373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43675 (local address 127.0.0.1:38960)
0620 12:18:17.999243 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:17.999256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:17.999354 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:17.999874 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:17.999877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:17.999895 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:17.999970 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.999975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.001266 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.001270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.002586 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:18.002601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.003794 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.003798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.003962 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.004024 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 12:18:18.004165 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 12:18:18.004378 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":86,"thread_start_us":95,"threads_started":1}
I20260620 12:18:18.004649 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.998589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.24.141.1:43675)
0620 12:18:17.998949 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.998956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.999388 (+ 432us) server_negotiation.cc:407] Connection header received
0620 12:18:17.999494 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.999499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.999699 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.999874 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:18.000101 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.001052 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.002789 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.003608 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.003735 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.003930 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:18.004237 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 12:18:18.004399 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":98,"threads_started":1}
I20260620 12:18:18.004649 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.998244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.24.141.3:34077)
0620 12:18:17.998739 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.998752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.999018 (+ 266us) server_negotiation.cc:407] Connection header received
0620 12:18:17.999193 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.999199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.999458 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.999611 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:17.999907 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.000915 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.002526 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.003439 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.003584 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.003914 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:18.004322 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 12:18:18.004442 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":155,"threads_started":1}
I20260620 12:18:18.005353 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.997098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.2:39531 (local address 127.0.0.1:44988)
0620 12:18:17.997575 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:17.997648 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:17.997929 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:17.998705 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:17.998713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:17.998745 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:17.998952 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.998961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.000758 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.000764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.002697 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:18.002706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.003971 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.003974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.004064 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.004084 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:18:18.004129 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 12:18:18.005015 (+ 886us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":936,"thread_start_us":123,"threads_started":1}
I20260620 12:18:18.005352 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.998006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34077 (local address 127.0.0.1:49990)
0620 12:18:17.998912 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:17.998929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:17.999047 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:17.999595 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:17.999599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:17.999616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:17.999772 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:17.999781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.001100 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.001103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.002372 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:18.002384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.004006 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.004009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.004100 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.004126 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:18:18.004206 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 12:18:18.005051 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":772,"thread_start_us":119,"threads_started":1}
I20260620 12:18:18.005751 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:17.997332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.24.141.2:39531)
0620 12:18:17.997839 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:17.997855 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:17.997889 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:18:17.998182 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:17.998189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:17.998493 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:17.999131 (+ 638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:17.999306 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.000551 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.003004 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.003805 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.004427 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.004823 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:18.004891 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:18:18.005578 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":220,"threads_started":1}
I20260620 12:18:18.008917 12562 tablet_service.cc:1467] Processing CreateTablet for tablet 28d1919792c34347b8a383a7fdd6e34b (DEFAULT_TABLE table=test-table [id=eb5767ac43ea4e1c98e3d48c7f85a8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:18.009155 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 28d1919792c34347b8a383a7fdd6e34b (DEFAULT_TABLE table=test-table [id=eb5767ac43ea4e1c98e3d48c7f85a8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:18.009155 12693 tablet_service.cc:1467] Processing CreateTablet for tablet 28d1919792c34347b8a383a7fdd6e34b (DEFAULT_TABLE table=test-table [id=eb5767ac43ea4e1c98e3d48c7f85a8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:18:18.010512 12693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d1919792c34347b8a383a7fdd6e34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:18.010512 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d1919792c34347b8a383a7fdd6e34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:18.010686 12562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d1919792c34347b8a383a7fdd6e34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:18.020234 12909 tablet_bootstrap.cc:492] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Bootstrap starting.
I20260620 12:18:18.020234 12908 tablet_bootstrap.cc:492] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Bootstrap starting.
I20260620 12:18:18.020757 12907 tablet_bootstrap.cc:492] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: Bootstrap starting.
I20260620 12:18:18.023028 12908 tablet_bootstrap.cc:654] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:18.023028 12909 tablet_bootstrap.cc:654] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:18.023075 12907 tablet_bootstrap.cc:654] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:18.024096 12909 log.cc:824] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:18.024096 12908 log.cc:824] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:18.024217 12907 log.cc:824] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:18.026917 12907 tablet_bootstrap.cc:492] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: No bootstrap required, opened a new log
I20260620 12:18:18.026919 12908 tablet_bootstrap.cc:492] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: No bootstrap required, opened a new log
I20260620 12:18:18.027240 12907 ts_tablet_manager.cc:1397] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:18:18.027251 12908 ts_tablet_manager.cc:1397] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:18:18.027499 12909 tablet_bootstrap.cc:492] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: No bootstrap required, opened a new log
I20260620 12:18:18.027751 12909 ts_tablet_manager.cc:1397] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 12:18:18.030987 12909 raft_consensus.cc:348] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.031296 12909 raft_consensus.cc:374] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:18.031345 12909 raft_consensus.cc:729] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef7be240631431eaca6dabafeb15828, State: Initialized, Role: FOLLOWER
I20260620 12:18:18.031839 12909 consensus_queue.cc:260] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.031977 12907 raft_consensus.cc:348] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.031875 12908 raft_consensus.cc:348] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.032235 12907 raft_consensus.cc:374] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:18.032236 12908 raft_consensus.cc:374] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:18.032294 12907 raft_consensus.cc:729] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfbb5ce7a710413d8ba70684952b4bc7, State: Initialized, Role: FOLLOWER
I20260620 12:18:18.032305 12908 raft_consensus.cc:729] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9ff04d558c4b1fa3983e863806abf4, State: Initialized, Role: FOLLOWER
I20260620 12:18:18.032902 12907 consensus_queue.cc:260] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.032902 12908 consensus_queue.cc:260] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.033049 12909 ts_tablet_manager.cc:1428] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:18.033725 12759 heartbeater.cc:503] Master 127.24.141.62:33393 was elected leader, sending a full tablet report...
I20260620 12:18:18.034207 12907 ts_tablet_manager.cc:1428] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:18:18.034207 12890 heartbeater.cc:503] Master 127.24.141.62:33393 was elected leader, sending a full tablet report...
I20260620 12:18:18.034255 12908 ts_tablet_manager.cc:1428] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:18:18.034653 12628 heartbeater.cc:503] Master 127.24.141.62:33393 was elected leader, sending a full tablet report...
W20260620 12:18:18.199052 12891 tablet.cc:2369] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:18:18.227183 12760 tablet.cc:2369] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:18:18.256373 12629 tablet.cc:2369] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:18:18.304011 12913 raft_consensus.cc:482] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:18:18.304241 12913 raft_consensus.cc:504] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.305475 12913 leader_election.cc:288] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9ff04d558c4b1fa3983e863806abf4 (127.24.141.3:34077), cfbb5ce7a710413d8ba70684952b4bc7 (127.24.141.1:43675)
I20260620 12:18:18.310456 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:18.306397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:49323 (local address 127.24.141.1:43675)
0620 12:18:18.306600 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:18.306606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:18.306700 (+ 94us) server_negotiation.cc:407] Connection header received
0620 12:18:18.306876 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:18.306879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:18.306938 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:18.307023 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:18.307481 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.308100 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.309341 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.309804 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.309938 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.310009 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:18.310205 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 12:18:18.310290 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 12:18:18.310436 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:18.305960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.141.2:60787 (local address 127.24.141.3:34077)
0620 12:18:18.306183 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:18.306188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:18.306214 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:18:18.306386 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:18.306391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:18.306456 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:18.306518 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:18:18.307094 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.307836 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.309242 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.309721 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.309849 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.309914 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:18.310148 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 12:18:18.310251 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 12:18:18.310505 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:18.305774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.3:34077 (local address 127.24.141.2:60787)
0620 12:18:18.306072 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:18.306089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:18.306197 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:18.306710 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:18.306713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:18.306759 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:18.306896 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:18.306904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.307994 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.307998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.309034 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:18.309049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.309878 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.309882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.310000 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.310026 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:18:18.310105 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:18:18.310215 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":36,"spinlock_wait_cycles":3712,"thread_start_us":111,"threads_started":1}
I20260620 12:18:18.310513 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:18.306202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43675 (local address 127.24.141.2:49323)
0620 12:18:18.306561 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:18.306577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:18.306755 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:18.307095 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:18.307098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:18.307126 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:18:18.307300 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:18.307308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.308252 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.308255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.309190 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:18.309204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.309961 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.309965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:18.310060 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.310081 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:18:18.310167 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:18:18.310250 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":2,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20260620 12:18:18.311475 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d1919792c34347b8a383a7fdd6e34b" candidate_uuid: "2ef7be240631431eaca6dabafeb15828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" is_pre_election: true
I20260620 12:18:18.311472 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d1919792c34347b8a383a7fdd6e34b" candidate_uuid: "2ef7be240631431eaca6dabafeb15828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9ff04d558c4b1fa3983e863806abf4" is_pre_election: true
I20260620 12:18:18.311918 12844 raft_consensus.cc:2454] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef7be240631431eaca6dabafeb15828 in term 0.
I20260620 12:18:18.311918 12582 raft_consensus.cc:2454] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef7be240631431eaca6dabafeb15828 in term 0.
I20260620 12:18:18.312655 12649 leader_election.cc:302] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9ff04d558c4b1fa3983e863806abf4, 2ef7be240631431eaca6dabafeb15828; no voters:
I20260620 12:18:18.313031 12913 raft_consensus.cc:2781] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:18:18.313215 12913 raft_consensus.cc:482] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:18:18.313275 12913 raft_consensus.cc:3037] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:18.315207 12913 raft_consensus.cc:504] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.315668 12913 leader_election.cc:288] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [CANDIDATE]: Term 1 election: Requested vote from peers 0d9ff04d558c4b1fa3983e863806abf4 (127.24.141.3:34077), cfbb5ce7a710413d8ba70684952b4bc7 (127.24.141.1:43675)
I20260620 12:18:18.316390 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d1919792c34347b8a383a7fdd6e34b" candidate_uuid: "2ef7be240631431eaca6dabafeb15828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbb5ce7a710413d8ba70684952b4bc7"
I20260620 12:18:18.316627 12582 raft_consensus.cc:3037] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:18.316869 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d1919792c34347b8a383a7fdd6e34b" candidate_uuid: "2ef7be240631431eaca6dabafeb15828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9ff04d558c4b1fa3983e863806abf4"
I20260620 12:18:18.317092 12844 raft_consensus.cc:3037] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:18.318521 12582 raft_consensus.cc:2454] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef7be240631431eaca6dabafeb15828 in term 1.
I20260620 12:18:18.318559 12844 raft_consensus.cc:2454] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef7be240631431eaca6dabafeb15828 in term 1.
I20260620 12:18:18.319068 12649 leader_election.cc:302] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9ff04d558c4b1fa3983e863806abf4, 2ef7be240631431eaca6dabafeb15828; no voters:
I20260620 12:18:18.319311 12913 raft_consensus.cc:2781] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:18.319587 12913 raft_consensus.cc:686] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 1 LEADER]: Becoming Leader. State: Replica: 2ef7be240631431eaca6dabafeb15828, State: Running, Role: LEADER
I20260620 12:18:18.320037 12913 consensus_queue.cc:237] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } }
I20260620 12:18:18.324735 12440 catalog_manager.cc:5495] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef7be240631431eaca6dabafeb15828 (127.24.141.2). New cstate: current_term: 1 leader_uuid: "2ef7be240631431eaca6dabafeb15828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef7be240631431eaca6dabafeb15828" member_type: VOTER last_known_addr { host: "127.24.141.2" port: 39531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20260620 12:18:18.385084 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:18.380109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.24.141.2:39531)
0620 12:18:18.380250 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:18.380253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:18.380270 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:18.380400 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:18.380403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:18.380524 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:18.380640 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:18.380928 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.381609 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.382933 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:18.383180 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:18.383282 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:18.383836 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:18.384508 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:18.384630 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:18.384801 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 12:18:18.384878 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:18:18.394481 12582 raft_consensus.cc:1261] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 1 FOLLOWER]: Refusing update from remote peer 2ef7be240631431eaca6dabafeb15828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:18.394482 12844 raft_consensus.cc:1261] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 1 FOLLOWER]: Refusing update from remote peer 2ef7be240631431eaca6dabafeb15828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:18.395422 12913 consensus_queue.cc:1035] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9ff04d558c4b1fa3983e863806abf4" member_type: VOTER last_known_addr { host: "127.24.141.3" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:18.395720 12918 consensus_queue.cc:1035] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfbb5ce7a710413d8ba70684952b4bc7" member_type: VOTER last_known_addr { host: "127.24.141.1" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:18.406154 12922 mvcc.cc:205] Tried to move back new op lower bound from 7298899551812734976 to 7298899551522492416. Current Snapshot: MvccSnapshot[applied={T|T < 7298899551812734976}]
I20260620 12:18:18.408493 12923 mvcc.cc:205] Tried to move back new op lower bound from 7298899551812734976 to 7298899551522492416. Current Snapshot: MvccSnapshot[applied={T|T < 7298899551812734976}]
I20260620 12:18:18.408740 12921 mvcc.cc:205] Tried to move back new op lower bound from 7298899551812734976 to 7298899551522492416. Current Snapshot: MvccSnapshot[applied={T|T < 7298899551812734976 or (T in {7298899551812734976})}]
I20260620 12:18:18.418339 12440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:18.418538 12440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:18:18.421674 12440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acb6a53aba044ed8884319728577bcc3: Sending DeleteTablet for 3 replicas of tablet 28d1919792c34347b8a383a7fdd6e34b
I20260620 12:18:18.422854 12693 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d1919792c34347b8a383a7fdd6e34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:18 UTC) from {username='slave'} at 127.0.0.1:44988
I20260620 12:18:18.422854 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d1919792c34347b8a383a7fdd6e34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:18 UTC) from {username='slave'} at 127.0.0.1:49990
I20260620 12:18:18.423154 12562 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d1919792c34347b8a383a7fdd6e34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:18:18 UTC) from {username='slave'} at 127.0.0.1:38960
I20260620 12:18:18.423548 12931 tablet_replica.cc:330] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: stopping tablet replica
I20260620 12:18:18.423954 12931 raft_consensus.cc:2229] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:18:18.424003 12932 tablet_replica.cc:330] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7: stopping tablet replica
I20260620 12:18:18.424278 12932 raft_consensus.cc:2229] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:18.424517 12931 raft_consensus.cc:2258] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:18.424604 12932 raft_consensus.cc:2258] T 28d1919792c34347b8a383a7fdd6e34b P cfbb5ce7a710413d8ba70684952b4bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:18.424631 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 12500
I20260620 12:18:18.424953 12930 tablet_replica.cc:330] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: stopping tablet replica
I20260620 12:18:18.425321 12930 raft_consensus.cc:2229] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:18:18.425606 12930 raft_consensus.cc:2258] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:18:18.428267 12930 ts_tablet_manager.cc:1905] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:18.428267 12931 ts_tablet_manager.cc:1905] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:18:18.432286 12930 ts_tablet_manager.cc:1918] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:18.432286 12931 ts_tablet_manager.cc:1918] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:18:18.432449 12931 log.cc:1196] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-1/wal/wals/28d1919792c34347b8a383a7fdd6e34b
I20260620 12:18:18.432449 12930 log.cc:1196] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Deleting WAL directory at /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957618728254-25140-0/minicluster-data/ts-2/wal/wals/28d1919792c34347b8a383a7fdd6e34b
I20260620 12:18:18.432791 12931 ts_tablet_manager.cc:1939] T 28d1919792c34347b8a383a7fdd6e34b P 2ef7be240631431eaca6dabafeb15828: Deleting consensus metadata
I20260620 12:18:18.432790 12930 ts_tablet_manager.cc:1939] T 28d1919792c34347b8a383a7fdd6e34b P 0d9ff04d558c4b1fa3983e863806abf4: Deleting consensus metadata
I20260620 12:18:18.434828 12427 catalog_manager.cc:4841] TS 2ef7be240631431eaca6dabafeb15828 (127.24.141.2:39531): tablet 28d1919792c34347b8a383a7fdd6e34b (table test-table [id=eb5767ac43ea4e1c98e3d48c7f85a8ae]) successfully deleted
I20260620 12:18:18.435070 12427 catalog_manager.cc:4841] TS 0d9ff04d558c4b1fa3983e863806abf4 (127.24.141.3:34077): tablet 28d1919792c34347b8a383a7fdd6e34b (table test-table [id=eb5767ac43ea4e1c98e3d48c7f85a8ae]) successfully deleted
W20260620 12:18:18.439419 12427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.141.1:43675 (error 108)
I20260620 12:18:18.440254 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 12631
I20260620 12:18:18.441798 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:18.441320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.1:43675 (local address 127.0.0.1:38972)
0620 12:18:18.441590 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:18.441676 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.1:43675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260620 12:18:18.442013 12427 catalog_manager.cc:4584] TS cfbb5ce7a710413d8ba70684952b4bc7 (127.24.141.1:43675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28d1919792c34347b8a383a7fdd6e34b: Network error: Client connection negotiation failed: client connection to 127.24.141.1:43675: connect: Connection refused (error 111)
I20260620 12:18:18.448292 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 12762
I20260620 12:18:18.454802 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 12409
2026-06-20T12:18:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1561 ms)
[----------] 6 tests from EncryptionPolicyTest (12114 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:18 dist-test-slave-rh5v krb5kdc[12936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:18 dist-test-slave-rh5v krb5kdc[12936](info): set up 2 sockets
Jun 20 12:18:18 dist-test-slave-rh5v krb5kdc[12936](info): commencing operation
krb5kdc: starting...
W20260620 12:18:19.429816 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:19 dist-test-slave-rh5v krb5kdc[12936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:19.576407 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:19.708488 12949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:19.708822 12949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:19.708885 12949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:19.713255 12949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:19.713366 12949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:19.713394 12949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:19.713439 12949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:19.713462 12949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:19.720739 12949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:19.722035 12949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:19.723150 12949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:19.732824 12957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:19.732824 12954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:19.732856 12955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:19.733229 12949 server_base.cc:1034] running on GCE node
I20260620 12:18:19.734027 12949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:19.735313 12949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:19.736510 12949 hybrid_clock.cc:648] HybridClock initialized: now 1781957899736474 us; error 61 us; skew 500 ppm
Jun 20 12:18:19 dist-test-slave-rh5v krb5kdc[12936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:19.739872 12949 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:19.741042 12949 webserver.cc:460] Webserver started at http://127.0.0.1:38805/ using document root <none> and password file <none>
I20260620 12:18:19.741716 12949 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:19.741789 12949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:19.742009 12949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:19.744262 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "b36a2bb0b23541fab5284c5dbe64c6f3"
format_stamp: "Formatted at 2026-06-20 12:18:19 on dist-test-slave-rh5v"
server_key: "c98adbc20c5e9f39e06c3bb58c3af687"
server_key_iv: "6084126851870fe5086bfcdd3ed23ff7"
server_key_version: "encryptionkey@0"
I20260620 12:18:19.744875 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "b36a2bb0b23541fab5284c5dbe64c6f3"
format_stamp: "Formatted at 2026-06-20 12:18:19 on dist-test-slave-rh5v"
server_key: "c98adbc20c5e9f39e06c3bb58c3af687"
server_key_iv: "6084126851870fe5086bfcdd3ed23ff7"
server_key_version: "encryptionkey@0"
I20260620 12:18:19.749286 12949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:19.752296 12964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:19.754248 12949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:18:19.754453 12949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "b36a2bb0b23541fab5284c5dbe64c6f3"
format_stamp: "Formatted at 2026-06-20 12:18:19 on dist-test-slave-rh5v"
server_key: "c98adbc20c5e9f39e06c3bb58c3af687"
server_key_iv: "6084126851870fe5086bfcdd3ed23ff7"
server_key_version: "encryptionkey@0"
I20260620 12:18:19.754612 12949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:19.777549 12949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:19.780951 12949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:19.781200 12949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:19.789048 12949 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36535
I20260620 12:18:19.789179 13016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36535 every 8 connection(s)
I20260620 12:18:19.790243 12949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:19.794090 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:19.795992 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 12949
I20260620 12:18:19.796147 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:19.796506 25140 external_mini_cluster.cc:1428] Setting key e3a0f1e82674b513ca46119fa610dcad
I20260620 12:18:19.802467 13017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3: Bootstrap starting.
I20260620 12:18:19.806120 13017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:19.807274 13017 log.cc:824] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:18:19 dist-test-slave-rh5v krb5kdc[12936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:18:19.810438 13017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3: No bootstrap required, opened a new log
I20260620 12:18:19.815835 13017 raft_consensus.cc:348] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260620 12:18:19.816188 13017 raft_consensus.cc:374] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:19.816238 13017 raft_consensus.cc:729] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36a2bb0b23541fab5284c5dbe64c6f3, State: Initialized, Role: FOLLOWER
I20260620 12:18:19.816435 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:19.798507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60819 (local address 127.0.0.1:36535)
0620 12:18:19.799050 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:19.799068 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:19.799108 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:18:19.799954 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:19.799976 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:19.800407 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:19.800741 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:19.802010 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:19.803391 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:19.804819 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:19.805637 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:19.805854 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:19.809294 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:19.809301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:19.809324 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:19.809363 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:19.811793 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:19.812431 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:19.812439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:19.812447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:19.812611 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:19.813398 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:19.813402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:19.813404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:19.814099 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:19.814387 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:19.814843 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 12:18:19.815171 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":171,"threads_started":1}
I20260620 12:18:19.816836 13017 consensus_queue.cc:260] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260620 12:18:19.816992 13017 raft_consensus.cc:388] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:19.817090 13017 raft_consensus.cc:482] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:19.817202 13017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:19.818367 13017 raft_consensus.cc:504] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260620 12:18:19.818877 13017 leader_election.cc:302] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b36a2bb0b23541fab5284c5dbe64c6f3; no voters:
I20260620 12:18:19.819312 13017 leader_election.cc:288] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:19.819473 13022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:19.819777 13022 raft_consensus.cc:686] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [term 1 LEADER]: Becoming Leader. State: Replica: b36a2bb0b23541fab5284c5dbe64c6f3, State: Running, Role: LEADER
I20260620 12:18:19.820170 13022 consensus_queue.cc:237] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260620 12:18:19.820829 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:19.821029 13017 sys_catalog.cc:564] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:19.822472 13023 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } } }
I20260620 12:18:19.822723 13023 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:19.823104 13024 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36a2bb0b23541fab5284c5dbe64c6f3. Latest consensus state: current_term: 1 leader_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36a2bb0b23541fab5284c5dbe64c6f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } } }
I20260620 12:18:19.823186 13035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:19.823199 13024 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:19.827854 13035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:19.835520 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:19.824165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43907 (local address 127.0.0.1:36535)
0620 12:18:19.824293 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:19.824299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:19.825438 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 12:18:19.825622 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:19.825626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:19.825683 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:19.825738 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:19.826779 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:19.827598 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:19.828842 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:19.829297 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:19.829457 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:19.831103 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:19.831106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:19.831108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:19.831140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:19.832612 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:19.833210 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:19.833216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:19.833218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:19.833406 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:19.834232 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:19.834236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:19.834238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:19.834522 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:19.834664 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:19.835158 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 12:18:19.835328 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 12:18:19.835603 13035 catalog_manager.cc:1347] Generated new cluster ID: b28614147e4f4d049a7ced8f4c26d760
I20260620 12:18:19.835917 13035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:19.849753 13035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:19.850950 13035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:19.858855 13035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b36a2bb0b23541fab5284c5dbe64c6f3: Generated new TSK 0
I20260620 12:18:19.859899 13035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:19.867939 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 12949
W20260620 12:18:19.875440 12934 connection.cc:535] client connection to 127.0.0.1:36535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36535): Connection reset by peer (error 104)
2026-06-20T12:18:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:19 dist-test-slave-rh5v krb5kdc[13048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:19 dist-test-slave-rh5v krb5kdc[13048](info): set up 2 sockets
Jun 20 12:18:19 dist-test-slave-rh5v krb5kdc[13048](info): commencing operation
krb5kdc: starting...
W20260620 12:18:20.841754 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:20 dist-test-slave-rh5v krb5kdc[13048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:20.980624 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:21.118943 13061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:21.119396 13061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:21.119468 13061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:21.124559 13061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:21.124703 13061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:21.124734 13061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:21.124758 13061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:21.124785 13061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:21.131450 13061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:21.133024 13061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:21.134392 13061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:21.145025 13067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:21.145105 13061 server_base.cc:1034] running on GCE node
W20260620 12:18:21.145140 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:21.145140 13066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:21.146219 13061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:21.148072 13061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:21.149307 13061 hybrid_clock.cc:648] HybridClock initialized: now 1781957901149258 us; error 53 us; skew 500 ppm
Jun 20 12:18:21 dist-test-slave-rh5v krb5kdc[13048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:21.152676 13061 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:21.153899 13061 webserver.cc:460] Webserver started at http://127.0.0.1:33595/ using document root <none> and password file <none>
I20260620 12:18:21.154567 13061 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:21.154646 13061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:21.154870 13061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:21.157501 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "efe28ee82141483280c15f93ae8ad92d"
format_stamp: "Formatted at 2026-06-20 12:18:21 on dist-test-slave-rh5v"
server_key: "ad72d94ab3c552e0885b0edb7db27029"
server_key_iv: "af47e444c8fa7f98bf32adbccfc8841d"
server_key_version: "encryptionkey@0"
I20260620 12:18:21.158147 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "efe28ee82141483280c15f93ae8ad92d"
format_stamp: "Formatted at 2026-06-20 12:18:21 on dist-test-slave-rh5v"
server_key: "ad72d94ab3c552e0885b0edb7db27029"
server_key_iv: "af47e444c8fa7f98bf32adbccfc8841d"
server_key_version: "encryptionkey@0"
I20260620 12:18:21.163316 13061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:21.166718 13076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:21.168602 13061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:21.168850 13061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "efe28ee82141483280c15f93ae8ad92d"
format_stamp: "Formatted at 2026-06-20 12:18:21 on dist-test-slave-rh5v"
server_key: "ad72d94ab3c552e0885b0edb7db27029"
server_key_iv: "af47e444c8fa7f98bf32adbccfc8841d"
server_key_version: "encryptionkey@0"
I20260620 12:18:21.169027 13061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:21.186350 13061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:21.189903 13061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:21.190097 13061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:21.198369 13061 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39103
I20260620 12:18:21.198415 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39103 every 8 connection(s)
I20260620 12:18:21.199738 13061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:21.203805 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:21.209985 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13061
I20260620 12:18:21.210166 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:21.210533 25140 external_mini_cluster.cc:1428] Setting key 8758f36099ef78caa27124f157985a03
I20260620 12:18:21.211880 13129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d: Bootstrap starting.
I20260620 12:18:21.215121 13129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:21.216323 13129 log.cc:824] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:21.219594 13129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d: No bootstrap required, opened a new log
I20260620 12:18:21.223754 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:21.212506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40513 (local address 127.0.0.1:39103)
0620 12:18:21.213103 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:21.213127 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:21.213508 (+ 381us) server_negotiation.cc:407] Connection header received
0620 12:18:21.214509 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:21.214532 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:21.215049 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:21.215446 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:21.216057 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:21.217466 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:21.218950 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:21.219746 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:21.219932 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:21.221118 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:21.221129 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:21.221149 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:21.221745 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:21.221828 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:21.221860 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:21.222005 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:21.222262 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 12:18:21.222600 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":162,"threads_started":1}
I20260620 12:18:21.224647 13129 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe28ee82141483280c15f93ae8ad92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260620 12:18:21.225042 13129 raft_consensus.cc:374] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:21.225121 13129 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe28ee82141483280c15f93ae8ad92d, State: Initialized, Role: FOLLOWER
I20260620 12:18:21.225687 13129 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe28ee82141483280c15f93ae8ad92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260620 12:18:21.225855 13129 raft_consensus.cc:388] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:21.225904 13129 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:21.226027 13129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:21.227442 13129 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe28ee82141483280c15f93ae8ad92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260620 12:18:21.227941 13129 leader_election.cc:302] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efe28ee82141483280c15f93ae8ad92d; no voters:
I20260620 12:18:21.228402 13129 leader_election.cc:288] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:21.228575 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:21.228564 13134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:21.228909 13134 raft_consensus.cc:686] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [term 1 LEADER]: Becoming Leader. State: Replica: efe28ee82141483280c15f93ae8ad92d, State: Running, Role: LEADER
I20260620 12:18:21.229441 13134 consensus_queue.cc:237] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe28ee82141483280c15f93ae8ad92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260620 12:18:21.230180 13129 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:21.231562 13136 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader efe28ee82141483280c15f93ae8ad92d. Latest consensus state: current_term: 1 leader_uuid: "efe28ee82141483280c15f93ae8ad92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe28ee82141483280c15f93ae8ad92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } } }
I20260620 12:18:21.231778 13136 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:21.232033 13135 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efe28ee82141483280c15f93ae8ad92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe28ee82141483280c15f93ae8ad92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } } }
I20260620 12:18:21.232144 13135 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:21.232300 13149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:21.237711 13149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:21.238408 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:21.232310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40713 (local address 127.0.0.1:39103)
0620 12:18:21.232444 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:21.232449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:21.232641 (+ 192us) server_negotiation.cc:407] Connection header received
0620 12:18:21.232873 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:21.232877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:21.232932 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:21.232984 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:21.233567 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:21.234285 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:21.235773 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:21.236168 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:21.236303 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:21.237761 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:21.237766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:21.237770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:21.237792 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:21.237834 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:21.237851 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:21.237959 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:21.238155 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 12:18:21.238215 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:18:21.244611 13149 catalog_manager.cc:1347] Generated new cluster ID: d26a15f318734b32b618f540e0d91db6
I20260620 12:18:21.244740 13149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:21.259657 13149 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:21.261461 13149 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:21.271606 13149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efe28ee82141483280c15f93ae8ad92d: Generated new TSK 0
I20260620 12:18:21.273003 13149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:21.281832 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13061
W20260620 12:18:21.288157 13046 connection.cc:535] client connection to 127.0.0.1:39103 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39103): Connection reset by peer (error 104)
2026-06-20T12:18:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:21 dist-test-slave-rh5v krb5kdc[13160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:21 dist-test-slave-rh5v krb5kdc[13160](info): set up 2 sockets
Jun 20 12:18:21 dist-test-slave-rh5v krb5kdc[13160](info): commencing operation
krb5kdc: starting...
W20260620 12:18:22.254061 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:22 dist-test-slave-rh5v krb5kdc[13160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:22.391853 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:22.538964 13173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:22.539511 13173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:22.539587 13173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:22.544093 13173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:22.544169 13173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:22.544189 13173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:22.544205 13173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:22.544220 13173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:22.550307 13173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:22.551816 13173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:22.553153 13173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:22.563861 13178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:22.563871 13179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:22.563822 13181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:22.564321 13173 server_base.cc:1034] running on GCE node
I20260620 12:18:22.565153 13173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:22.566730 13173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:22.567915 13173 hybrid_clock.cc:648] HybridClock initialized: now 1781957902567889 us; error 65 us; skew 500 ppm
Jun 20 12:18:22 dist-test-slave-rh5v krb5kdc[13160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:22.571441 13173 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:22.572667 13173 webserver.cc:460] Webserver started at http://127.0.0.1:37253/ using document root <none> and password file <none>
I20260620 12:18:22.573403 13173 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:22.573495 13173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:22.573729 13173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:22.576211 13173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "fdc7077eee874c648c5a47944dc8019f"
format_stamp: "Formatted at 2026-06-20 12:18:22 on dist-test-slave-rh5v"
server_key: "788aadcdcd1cb318337153a4908316bc"
server_key_iv: "76fc9db9bfc1f74c76911021707b0e89"
server_key_version: "encryptionkey@0"
I20260620 12:18:22.576855 13173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "fdc7077eee874c648c5a47944dc8019f"
format_stamp: "Formatted at 2026-06-20 12:18:22 on dist-test-slave-rh5v"
server_key: "788aadcdcd1cb318337153a4908316bc"
server_key_iv: "76fc9db9bfc1f74c76911021707b0e89"
server_key_version: "encryptionkey@0"
I20260620 12:18:22.581331 13173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:18:22.584383 13188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:22.586220 13173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:18:22.586380 13173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "fdc7077eee874c648c5a47944dc8019f"
format_stamp: "Formatted at 2026-06-20 12:18:22 on dist-test-slave-rh5v"
server_key: "788aadcdcd1cb318337153a4908316bc"
server_key_iv: "76fc9db9bfc1f74c76911021707b0e89"
server_key_version: "encryptionkey@0"
I20260620 12:18:22.586515 13173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:22.592062 13173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:22.596275 13173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:22.596560 13173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:22.605640 13173 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37775
I20260620 12:18:22.605677 13240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37775 every 8 connection(s)
I20260620 12:18:22.606899 13173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:22.610510 13241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:22.610843 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13173
I20260620 12:18:22.610997 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:22.611317 25140 external_mini_cluster.cc:1428] Setting key 52a087e7e7369932195b798ebaa93c96
I20260620 12:18:22.618296 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:22.613125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56289 (local address 127.0.0.1:37775)
0620 12:18:22.613663 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:22.613683 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:22.613734 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:18:22.614654 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:22.614678 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:22.615095 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:22.615484 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:22.616574 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:22.616580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:22.616595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:22.616608 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:22.616682 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:22.616707 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:22.616858 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:22.617032 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 12:18:22.617299 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":156,"threads_started":1}
I20260620 12:18:22.618322 13241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f: Bootstrap starting.
I20260620 12:18:22.621788 13241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:22.622383 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:22.622749 13241 log.cc:824] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:22.625725 13241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f: No bootstrap required, opened a new log
I20260620 12:18:22.627022 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:22.625120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36147 (local address 127.0.0.1:37775)
0620 12:18:22.625288 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:22.625293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:22.625443 (+ 150us) server_negotiation.cc:407] Connection header received
0620 12:18:22.625612 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:22.625615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:22.625662 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:22.625774 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:22.626571 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:22.626575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:22.626577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:22.626591 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:22.626626 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:22.626640 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:22.626776 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:22.626817 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 12:18:22.626878 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:18:22.629993 13241 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc7077eee874c648c5a47944dc8019f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260620 12:18:22.630327 13241 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:22.630373 13241 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc7077eee874c648c5a47944dc8019f, State: Initialized, Role: FOLLOWER
I20260620 12:18:22.630863 13241 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc7077eee874c648c5a47944dc8019f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260620 12:18:22.631001 13241 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:22.631067 13241 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:22.631170 13241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:22.632287 13241 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc7077eee874c648c5a47944dc8019f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260620 12:18:22.632704 13241 leader_election.cc:302] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc7077eee874c648c5a47944dc8019f; no voters:
I20260620 12:18:22.633113 13241 leader_election.cc:288] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:22.633297 13251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:22.633666 13251 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [term 1 LEADER]: Becoming Leader. State: Replica: fdc7077eee874c648c5a47944dc8019f, State: Running, Role: LEADER
I20260620 12:18:22.634197 13251 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc7077eee874c648c5a47944dc8019f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260620 12:18:22.634943 13241 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:22.636315 13252 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc7077eee874c648c5a47944dc8019f. Latest consensus state: current_term: 1 leader_uuid: "fdc7077eee874c648c5a47944dc8019f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc7077eee874c648c5a47944dc8019f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } } }
I20260620 12:18:22.636462 13252 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:22.636744 13253 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdc7077eee874c648c5a47944dc8019f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc7077eee874c648c5a47944dc8019f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } } }
I20260620 12:18:22.636904 13253 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:22.636902 13260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:22.641448 13260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:22.649190 13260 catalog_manager.cc:1347] Generated new cluster ID: ef6ad9b162ea4deb9473682395d7af93
I20260620 12:18:22.649317 13260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:22.674525 13260 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:22.675037 13260 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:22.683962 13260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdc7077eee874c648c5a47944dc8019f: Generated new TSK 0
I20260620 12:18:22.684963 13260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:22.700639 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13173
2026-06-20T12:18:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:22 dist-test-slave-rh5v krb5kdc[13272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:22 dist-test-slave-rh5v krb5kdc[13272](info): set up 2 sockets
Jun 20 12:18:22 dist-test-slave-rh5v krb5kdc[13272](info): commencing operation
krb5kdc: starting...
W20260620 12:18:23.685757 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:23 dist-test-slave-rh5v krb5kdc[13272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:23.824645 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:23.953895 13285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:23.954231 13285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:23.954275 13285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:23.958603 13285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:23.958703 13285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:23.958724 13285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:23.958740 13285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:23.958755 13285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:23.965265 13285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:23.966684 13285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:23.967882 13285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:23.977674 13291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:23.977682 13290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:23.977674 13293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:23.978119 13285 server_base.cc:1034] running on GCE node
I20260620 12:18:23.978816 13285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:23.980140 13285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:23.981362 13285 hybrid_clock.cc:648] HybridClock initialized: now 1781957903981324 us; error 60 us; skew 500 ppm
Jun 20 12:18:23 dist-test-slave-rh5v krb5kdc[13272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:23.988142 13285 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:23.989410 13285 webserver.cc:460] Webserver started at http://127.0.0.1:36775/ using document root <none> and password file <none>
I20260620 12:18:23.990057 13285 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:23.990135 13285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:23.990351 13285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:23.992676 13285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9"
format_stamp: "Formatted at 2026-06-20 12:18:23 on dist-test-slave-rh5v"
server_key: "fb176c4be35e8ec75761aaa23e8a7ecc"
server_key_iv: "1fd4ae623ebfc1b4386da11141f8437b"
server_key_version: "encryptionkey@0"
I20260620 12:18:23.993369 13285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9"
format_stamp: "Formatted at 2026-06-20 12:18:23 on dist-test-slave-rh5v"
server_key: "fb176c4be35e8ec75761aaa23e8a7ecc"
server_key_iv: "1fd4ae623ebfc1b4386da11141f8437b"
server_key_version: "encryptionkey@0"
I20260620 12:18:23.998059 13285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:18:24.001507 13300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:24.003319 13285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:18:24.003518 13285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9"
format_stamp: "Formatted at 2026-06-20 12:18:23 on dist-test-slave-rh5v"
server_key: "fb176c4be35e8ec75761aaa23e8a7ecc"
server_key_iv: "1fd4ae623ebfc1b4386da11141f8437b"
server_key_version: "encryptionkey@0"
I20260620 12:18:24.003656 13285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:24.046283 13285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:24.049782 13285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:24.050000 13285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:24.058326 13285 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45613
I20260620 12:18:24.058370 13352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45613 every 8 connection(s)
I20260620 12:18:24.059497 13285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:24.063493 13353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:24.064917 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13285
I20260620 12:18:24.065110 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:24.065448 25140 external_mini_cluster.cc:1428] Setting key d13d4661c974a4ed7d4b808814a054e6
I20260620 12:18:24.071543 13353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9: Bootstrap starting.
I20260620 12:18:24.074877 13353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:24.075974 13353 log.cc:824] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:18:24 dist-test-slave-rh5v krb5kdc[13272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:18:24.078694 13353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9: No bootstrap required, opened a new log
I20260620 12:18:24.082911 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:24.067332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51063 (local address 127.0.0.1:45613)
0620 12:18:24.067862 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:24.067880 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:24.067921 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:18:24.068757 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:24.068779 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:24.069276 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:24.069624 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:24.070773 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:24.071942 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:24.073242 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:24.073825 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:24.073993 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:24.077353 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:24.077361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:24.077383 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:24.077422 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:24.079332 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:24.079954 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:24.079961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:24.079967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:24.080135 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:24.080449 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:24.080451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:24.080452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:24.080947 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:24.081191 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:24.081583 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 12:18:24.081991 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":166,"threads_started":1}
I20260620 12:18:24.083321 13353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260620 12:18:24.083622 13353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:24.083683 13353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8de6cfca0542c8aeb5d05a1fb00fa9, State: Initialized, Role: FOLLOWER
I20260620 12:18:24.084205 13353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260620 12:18:24.084362 13353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:24.084438 13353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:24.084545 13353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:24.085868 13353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260620 12:18:24.086277 13353 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8de6cfca0542c8aeb5d05a1fb00fa9; no voters:
I20260620 12:18:24.086509 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:24.086706 13353 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:24.086861 13358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:24.087141 13358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [term 1 LEADER]: Becoming Leader. State: Replica: 7d8de6cfca0542c8aeb5d05a1fb00fa9, State: Running, Role: LEADER
I20260620 12:18:24.087656 13358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260620 12:18:24.088276 13353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:24.089661 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8de6cfca0542c8aeb5d05a1fb00fa9. Latest consensus state: current_term: 1 leader_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } } }
I20260620 12:18:24.089898 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:24.090101 13359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8de6cfca0542c8aeb5d05a1fb00fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } } }
I20260620 12:18:24.090179 13359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:24.090548 13374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:24.094816 13374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:24.100692 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:24.089286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38463 (local address 127.0.0.1:45613)
0620 12:18:24.089449 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:24.089454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:24.090711 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 12:18:24.091066 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:24.091070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:24.091138 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:24.091238 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:24.092209 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:24.092812 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:24.093920 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:24.094271 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:24.094382 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:24.096569 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:24.096573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:24.096577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:24.096613 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:24.098430 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:24.099115 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:24.099120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:24.099122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:24.099230 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:24.099616 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:24.099620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:24.099622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:24.099891 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:24.100036 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:24.100309 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 12:18:24.100478 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:18:24.104946 13374 catalog_manager.cc:1347] Generated new cluster ID: 5b2342afd7844097882043b4403ecad7
I20260620 12:18:24.105103 13374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:24.129853 13374 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:24.131477 13374 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:24.140316 13374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8de6cfca0542c8aeb5d05a1fb00fa9: Generated new TSK 0
I20260620 12:18:24.141474 13374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:24.148500 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13285
2026-06-20T12:18:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:24 dist-test-slave-rh5v krb5kdc[13384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:24 dist-test-slave-rh5v krb5kdc[13384](info): set up 2 sockets
Jun 20 12:18:24 dist-test-slave-rh5v krb5kdc[13384](info): commencing operation
krb5kdc: starting...
W20260620 12:18:25.105890 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:25 dist-test-slave-rh5v krb5kdc[13384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:25.249279 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:25.382665 13397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:25.383020 13397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:25.383066 13397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:25.387564 13397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:25.387662 13397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:25.387682 13397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:25.387698 13397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:25.387713 13397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:25.394932 13397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:25.396804 13397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:25.398102 13397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:25.408844 13402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:25.408869 13405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:25.408880 13403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:25.409129 13397 server_base.cc:1034] running on GCE node
I20260620 12:18:25.410049 13397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:25.411465 13397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:25.412724 13397 hybrid_clock.cc:648] HybridClock initialized: now 1781957905412681 us; error 65 us; skew 500 ppm
Jun 20 12:18:25 dist-test-slave-rh5v krb5kdc[13384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:25.416776 13397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:25.418303 13397 webserver.cc:460] Webserver started at http://127.0.0.1:42581/ using document root <none> and password file <none>
I20260620 12:18:25.419009 13397 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:25.419093 13397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:25.419318 13397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:25.421814 13397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "729ee7e0a1ab4ed7987162b949c0ee07"
format_stamp: "Formatted at 2026-06-20 12:18:25 on dist-test-slave-rh5v"
server_key: "58b45e318634865f79839f10867c24fe"
server_key_iv: "9c10a028b3c5c76a5afcf8b6de513743"
server_key_version: "encryptionkey@0"
I20260620 12:18:25.422490 13397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "729ee7e0a1ab4ed7987162b949c0ee07"
format_stamp: "Formatted at 2026-06-20 12:18:25 on dist-test-slave-rh5v"
server_key: "58b45e318634865f79839f10867c24fe"
server_key_iv: "9c10a028b3c5c76a5afcf8b6de513743"
server_key_version: "encryptionkey@0"
I20260620 12:18:25.427755 13397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:25.432202 13412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:25.434594 13397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:25.434857 13397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "729ee7e0a1ab4ed7987162b949c0ee07"
format_stamp: "Formatted at 2026-06-20 12:18:25 on dist-test-slave-rh5v"
server_key: "58b45e318634865f79839f10867c24fe"
server_key_iv: "9c10a028b3c5c76a5afcf8b6de513743"
server_key_version: "encryptionkey@0"
I20260620 12:18:25.435051 13397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:25.449728 13397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:25.453862 13397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:25.454090 13397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:25.464428 13397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39741
I20260620 12:18:25.464459 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39741 every 8 connection(s)
I20260620 12:18:25.466353 13397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:25.468561 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13397
I20260620 12:18:25.468750 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:25.469187 25140 external_mini_cluster.cc:1428] Setting key 729e741bac1eac7553a9b53aac560ed4
I20260620 12:18:25.471161 13465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:25.479187 13465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07: Bootstrap starting.
I20260620 12:18:25.482661 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:25.471942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47335 (local address 127.0.0.1:39741)
0620 12:18:25.472584 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:25.472605 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:25.472656 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:18:25.473580 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:25.473606 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:25.474134 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:25.474574 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:25.475055 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:25.476231 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:25.477671 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:25.478309 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:25.478502 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:25.479600 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:25.479610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:25.479629 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:25.480186 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:25.480279 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:25.480317 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:25.480446 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:25.480746 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 12:18:25.481170 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":187,"threads_started":1}
I20260620 12:18:25.482923 13465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:25.484251 13465 log.cc:824] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:25.487587 13465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07: No bootstrap required, opened a new log
I20260620 12:18:25.487736 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:25.492663 13465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260620 12:18:25.493206 13465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:25.493280 13465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729ee7e0a1ab4ed7987162b949c0ee07, State: Initialized, Role: FOLLOWER
I20260620 12:18:25.493872 13465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260620 12:18:25.494065 13465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:25.494127 13465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:25.494248 13465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:25.495577 13465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260620 12:18:25.496119 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:25.490785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36733 (local address 127.0.0.1:39741)
0620 12:18:25.490985 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:25.490991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:25.491115 (+ 124us) server_negotiation.cc:407] Connection header received
0620 12:18:25.491257 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:25.491261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:25.491323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:25.491458 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:25.491877 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:25.492659 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:25.493897 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:25.494271 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:25.494424 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:25.495554 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:25.495559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:25.495562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:25.495581 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:25.495623 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:25.495638 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:25.495729 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:25.495889 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 12:18:25.495968 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 12:18:25.496070 13465 leader_election.cc:302] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729ee7e0a1ab4ed7987162b949c0ee07; no voters:
I20260620 12:18:25.496513 13465 leader_election.cc:288] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:25.496716 13475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:25.497020 13475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [term 1 LEADER]: Becoming Leader. State: Replica: 729ee7e0a1ab4ed7987162b949c0ee07, State: Running, Role: LEADER
I20260620 12:18:25.497476 13475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260620 12:18:25.498605 13465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:25.499300 13475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729ee7e0a1ab4ed7987162b949c0ee07. Latest consensus state: current_term: 1 leader_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } } }
I20260620 12:18:25.499482 13475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:25.499706 13476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729ee7e0a1ab4ed7987162b949c0ee07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } } }
I20260620 12:18:25.499879 13476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:25.499936 13479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:25.504184 13479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:25.512439 13479 catalog_manager.cc:1347] Generated new cluster ID: f6cfa2ddb79d46c9834b59b62c9f51f0
I20260620 12:18:25.512554 13479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:25.523370 13479 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:25.524489 13479 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:25.531641 13479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 729ee7e0a1ab4ed7987162b949c0ee07: Generated new TSK 0
I20260620 12:18:25.532449 13479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:25.546414 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13397
2026-06-20T12:18:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:25 dist-test-slave-rh5v krb5kdc[13496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:25 dist-test-slave-rh5v krb5kdc[13496](info): set up 2 sockets
Jun 20 12:18:25 dist-test-slave-rh5v krb5kdc[13496](info): commencing operation
krb5kdc: starting...
W20260620 12:18:26.533634 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:26 dist-test-slave-rh5v krb5kdc[13496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:26.676205 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:26.811427 13509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:26.811848 13509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:26.811897 13509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:26.816458 13509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:26.816574 13509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:26.816599 13509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:26.816618 13509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:26.816633 13509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:26.822921 13509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:26.824455 13509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:26.825819 13509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:26.836941 13517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:26.836954 13515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:26.836969 13514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:26.837014 13509 server_base.cc:1034] running on GCE node
I20260620 12:18:26.838181 13509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:26.839773 13509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:26.841039 13509 hybrid_clock.cc:648] HybridClock initialized: now 1781957906841032 us; error 47 us; skew 500 ppm
Jun 20 12:18:26 dist-test-slave-rh5v krb5kdc[13496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:26.844656 13509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:26.845992 13509 webserver.cc:460] Webserver started at http://127.0.0.1:44713/ using document root <none> and password file <none>
I20260620 12:18:26.846724 13509 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:26.846801 13509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:26.847049 13509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:26.849845 13509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "eb62cb8ebd5b4609be39b67c36a210f0"
format_stamp: "Formatted at 2026-06-20 12:18:26 on dist-test-slave-rh5v"
server_key: "e935522ba99bf08c189cd78108294001"
server_key_iv: "adbd9ab5fef54890a5fcd8c0506ec8e9"
server_key_version: "encryptionkey@0"
I20260620 12:18:26.850641 13509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "eb62cb8ebd5b4609be39b67c36a210f0"
format_stamp: "Formatted at 2026-06-20 12:18:26 on dist-test-slave-rh5v"
server_key: "e935522ba99bf08c189cd78108294001"
server_key_iv: "adbd9ab5fef54890a5fcd8c0506ec8e9"
server_key_version: "encryptionkey@0"
I20260620 12:18:26.856266 13509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:26.860203 13524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:26.862478 13509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:26.862741 13509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "eb62cb8ebd5b4609be39b67c36a210f0"
format_stamp: "Formatted at 2026-06-20 12:18:26 on dist-test-slave-rh5v"
server_key: "e935522ba99bf08c189cd78108294001"
server_key_iv: "adbd9ab5fef54890a5fcd8c0506ec8e9"
server_key_version: "encryptionkey@0"
I20260620 12:18:26.862922 13509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:26.869995 13509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:26.874475 13509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:26.874747 13509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:26.884543 13509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34055
I20260620 12:18:26.884582 13576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34055 every 8 connection(s)
I20260620 12:18:26.885921 13509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:26.886232 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13509
I20260620 12:18:26.886384 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:26.886732 25140 external_mini_cluster.cc:1428] Setting key c31f780183b1daa632b6fdab22036a2b
I20260620 12:18:26.890148 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:26.893855 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:26.888381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34649 (local address 127.0.0.1:34055)
0620 12:18:26.888964 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:26.888983 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:26.889030 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:18:26.889848 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:26.889872 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:26.890264 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:26.890672 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:26.891881 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:26.891886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:26.891902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:26.891916 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:26.891989 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:26.892013 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:26.892170 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:26.892295 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 12:18:26.892463 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":159,"threads_started":1}
I20260620 12:18:26.898514 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:26.899396 13577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0: Bootstrap starting.
I20260620 12:18:26.903123 13577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:26.903716 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:26.901640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42503 (local address 127.0.0.1:34055)
0620 12:18:26.901836 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:26.901842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:26.902110 (+ 268us) server_negotiation.cc:407] Connection header received
0620 12:18:26.902259 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:26.902263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:26.902304 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:26.902376 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:26.903133 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:26.903137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:26.903140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:26.903154 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:26.903182 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:26.903194 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:26.903300 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:26.903540 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 12:18:26.903581 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:18:26.904294 13577 log.cc:824] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:26.907446 13577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0: No bootstrap required, opened a new log
I20260620 12:18:26.912241 13577 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 12:18:26.912643 13577 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:26.912703 13577 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb62cb8ebd5b4609be39b67c36a210f0, State: Initialized, Role: FOLLOWER
I20260620 12:18:26.913395 13577 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 12:18:26.913648 13577 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:26.913730 13577 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:26.913861 13577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:26.915396 13577 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 12:18:26.915913 13577 leader_election.cc:302] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb62cb8ebd5b4609be39b67c36a210f0; no voters:
I20260620 12:18:26.916368 13577 leader_election.cc:288] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:26.916621 13587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:26.917027 13587 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [term 1 LEADER]: Becoming Leader. State: Replica: eb62cb8ebd5b4609be39b67c36a210f0, State: Running, Role: LEADER
I20260620 12:18:26.917603 13587 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 12:18:26.918213 13577 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:26.919734 13589 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb62cb8ebd5b4609be39b67c36a210f0. Latest consensus state: current_term: 1 leader_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } } }
I20260620 12:18:26.919970 13589 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:26.919975 13588 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb62cb8ebd5b4609be39b67c36a210f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } } }
I20260620 12:18:26.920372 13588 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:26.920531 13597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:26.925187 13597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:26.933135 13597 catalog_manager.cc:1347] Generated new cluster ID: 548a2a949f09445fa277d51bc82d2adb
I20260620 12:18:26.933267 13597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:26.955044 13597 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:26.955560 13597 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:26.960837 13597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb62cb8ebd5b4609be39b67c36a210f0: Generated new TSK 0
I20260620 12:18:26.962036 13597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:26.970006 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13509
2026-06-20T12:18:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:27 dist-test-slave-rh5v krb5kdc[13608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:27 dist-test-slave-rh5v krb5kdc[13608](info): set up 2 sockets
Jun 20 12:18:27 dist-test-slave-rh5v krb5kdc[13608](info): commencing operation
krb5kdc: starting...
W20260620 12:18:27.957881 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:28 dist-test-slave-rh5v krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:28.093953 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:28.228051 13621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:28.228466 13621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:28.228544 13621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:28.233217 13621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:28.233318 13621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:28.233350 13621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:28.233373 13621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:28.233464 13621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:28.239910 13621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:28.241418 13621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:28.242694 13621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:28.253123 13626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:28.253145 13629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:28.253317 13621 server_base.cc:1034] running on GCE node
W20260620 12:18:28.253123 13627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:28.254407 13621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:28.255920 13621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:28.257158 13621 hybrid_clock.cc:648] HybridClock initialized: now 1781957908257131 us; error 51 us; skew 500 ppm
Jun 20 12:18:28 dist-test-slave-rh5v krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:28.260921 13621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:28.262279 13621 webserver.cc:460] Webserver started at http://127.0.0.1:46257/ using document root <none> and password file <none>
I20260620 12:18:28.263044 13621 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:28.263128 13621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:28.263372 13621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:28.266218 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "ac78ac82e48c487f9d75a8b4a44d7283"
format_stamp: "Formatted at 2026-06-20 12:18:28 on dist-test-slave-rh5v"
server_key: "d4377f2c7c7bdf2b96e5866e8450592a"
server_key_iv: "c8d1c8ee1869f331606bd44d96dd1e4f"
server_key_version: "encryptionkey@0"
I20260620 12:18:28.266907 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "ac78ac82e48c487f9d75a8b4a44d7283"
format_stamp: "Formatted at 2026-06-20 12:18:28 on dist-test-slave-rh5v"
server_key: "d4377f2c7c7bdf2b96e5866e8450592a"
server_key_iv: "c8d1c8ee1869f331606bd44d96dd1e4f"
server_key_version: "encryptionkey@0"
I20260620 12:18:28.272176 13621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:28.276027 13636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:28.278095 13621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:28.278342 13621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "ac78ac82e48c487f9d75a8b4a44d7283"
format_stamp: "Formatted at 2026-06-20 12:18:28 on dist-test-slave-rh5v"
server_key: "d4377f2c7c7bdf2b96e5866e8450592a"
server_key_iv: "c8d1c8ee1869f331606bd44d96dd1e4f"
server_key_version: "encryptionkey@0"
I20260620 12:18:28.278517 13621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:28.297397 13621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:28.301877 13621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:28.302117 13621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:28.311067 13621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43587
I20260620 12:18:28.311106 13688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43587 every 8 connection(s)
I20260620 12:18:28.312328 13621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:28.312839 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13621
I20260620 12:18:28.313002 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:28.313308 25140 external_mini_cluster.cc:1428] Setting key fe1d55065651f501bccfac44ae7a7300
I20260620 12:18:28.316306 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:28.324661 13689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283: Bootstrap starting.
Jun 20 12:18:28 dist-test-slave-rh5v krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:18:28.328037 13689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:28.329092 13689 log.cc:824] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:28.332115 13689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283: No bootstrap required, opened a new log
I20260620 12:18:28.332427 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:28.315232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:43787 (local address 127.0.0.1:43587)
0620 12:18:28.315824 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:28.315857 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:28.315898 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:18:28.316697 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:28.316727 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:28.317252 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:28.317605 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:28.318792 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:28.319987 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:28.321437 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:28.322108 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:28.322269 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:28.325834 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:28.325842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:28.325877 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:28.325933 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:28.328065 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:28.328872 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:28.328881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:28.328888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:28.329159 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:28.329616 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:28.329621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:28.329623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:28.330196 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:28.330492 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:28.330940 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 12:18:28.331233 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":174,"threads_started":1}
I20260620 12:18:28.337121 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:28.337013 13689 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260620 12:18:28.337405 13689 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:28.337462 13689 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac78ac82e48c487f9d75a8b4a44d7283, State: Initialized, Role: FOLLOWER
I20260620 12:18:28.338039 13689 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260620 12:18:28.338227 13689 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:28.338295 13689 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:28.338405 13689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:28.339874 13689 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260620 12:18:28.340308 13689 leader_election.cc:302] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac78ac82e48c487f9d75a8b4a44d7283; no voters:
I20260620 12:18:28.340647 13689 leader_election.cc:288] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:28.340750 13699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:28.340945 13699 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [term 1 LEADER]: Becoming Leader. State: Replica: ac78ac82e48c487f9d75a8b4a44d7283, State: Running, Role: LEADER
I20260620 12:18:28.341439 13699 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260620 12:18:28.342361 13689 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:28.342886 13699 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac78ac82e48c487f9d75a8b4a44d7283. Latest consensus state: current_term: 1 leader_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } } }
I20260620 12:18:28.343735 13699 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:28.343606 13700 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78ac82e48c487f9d75a8b4a44d7283" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } } }
I20260620 12:18:28.344095 13700 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283 [sys.catalog]: This master's current role is: LEADER
W20260620 12:18:28.348958 13714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:28.349089 13714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:28.349187 13708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:28.350174 13708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:28.350376 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:28.340065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:47773 (local address 127.0.0.1:43587)
0620 12:18:28.340255 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:28.340260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:28.340510 (+ 250us) server_negotiation.cc:407] Connection header received
0620 12:18:28.340668 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:28.340672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:28.340733 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:28.340811 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:28.341805 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:28.342367 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:28.343443 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:28.343783 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:28.343863 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:28.346693 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:28.346697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:28.346700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:28.346731 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:28.348230 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:28.348798 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:28.348802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:28.348803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:28.348881 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:28.349274 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:28.349278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:28.349279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:28.349494 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:28.349637 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:28.349934 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 12:18:28.350142 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:18:28.358551 13708 catalog_manager.cc:1347] Generated new cluster ID: 7324c95897b141c2b3aefcb6e8c9fdd6
I20260620 12:18:28.358739 13708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:28.375908 13708 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:28.377315 13708 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:28.385183 13708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac78ac82e48c487f9d75a8b4a44d7283: Generated new TSK 0
I20260620 12:18:28.386029 13708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:28.393232 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13621
W20260620 12:18:28.400028 13606 connection.cc:535] client connection to 127.0.0.1:43587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43587): Connection reset by peer (error 104)
2026-06-20T12:18:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:28 dist-test-slave-rh5v krb5kdc[13720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:28 dist-test-slave-rh5v krb5kdc[13720](info): set up 2 sockets
Jun 20 12:18:28 dist-test-slave-rh5v krb5kdc[13720](info): commencing operation
krb5kdc: starting...
W20260620 12:18:29.370014 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:29 dist-test-slave-rh5v krb5kdc[13720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:29.506258 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:29.634250 13733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:29.634605 13733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:29.634670 13733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:29.639039 13733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:29.639133 13733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:29.639161 13733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:29.639183 13733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:29.639205 13733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:29.646111 13733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:29.647753 13733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:29.649159 13733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:29.659602 13739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:29.659608 13741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:29.659602 13738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:29.660116 13733 server_base.cc:1034] running on GCE node
I20260620 12:18:29.660929 13733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:29.662591 13733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:29.663807 13733 hybrid_clock.cc:648] HybridClock initialized: now 1781957909663779 us; error 63 us; skew 500 ppm
Jun 20 12:18:29 dist-test-slave-rh5v krb5kdc[13720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:29.672698 13733 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:29.674048 13733 webserver.cc:460] Webserver started at http://127.0.0.1:42755/ using document root <none> and password file <none>
I20260620 12:18:29.674791 13733 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:29.674868 13733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:29.675117 13733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:29.677815 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "bfe42b53a56949b59892b9953af30b96"
format_stamp: "Formatted at 2026-06-20 12:18:29 on dist-test-slave-rh5v"
server_key: "19d244731518098537c1a0fd5ac4bfdb"
server_key_iv: "c9af53c54608736191415e7aaa4b614d"
server_key_version: "encryptionkey@0"
I20260620 12:18:29.678476 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "bfe42b53a56949b59892b9953af30b96"
format_stamp: "Formatted at 2026-06-20 12:18:29 on dist-test-slave-rh5v"
server_key: "19d244731518098537c1a0fd5ac4bfdb"
server_key_iv: "c9af53c54608736191415e7aaa4b614d"
server_key_version: "encryptionkey@0"
I20260620 12:18:29.683391 13733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:29.686628 13748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:29.688467 13733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:18:29.688652 13733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "bfe42b53a56949b59892b9953af30b96"
format_stamp: "Formatted at 2026-06-20 12:18:29 on dist-test-slave-rh5v"
server_key: "19d244731518098537c1a0fd5ac4bfdb"
server_key_iv: "c9af53c54608736191415e7aaa4b614d"
server_key_version: "encryptionkey@0"
I20260620 12:18:29.688788 13733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:29.705222 13733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:29.709028 13733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:29.709297 13733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:29.718345 13733 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40839
I20260620 12:18:29.718370 13800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40839 every 8 connection(s)
I20260620 12:18:29.719556 13733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:29.723009 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:29.725677 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13733
I20260620 12:18:29.725839 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:29.726155 25140 external_mini_cluster.cc:1428] Setting key 33f86e593f3223af1deb8ad770ee95f1
I20260620 12:18:29.730132 13801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96: Bootstrap starting.
I20260620 12:18:29.733144 13801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:29.734099 13801 log.cc:824] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:29.736835 13801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96: No bootstrap required, opened a new log
I20260620 12:18:29.737695 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:29.728086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:58379 (local address 127.0.0.1:40839)
0620 12:18:29.728642 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:29.728663 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:29.728710 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:18:29.729647 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:29.729673 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:29.730159 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:29.730571 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:29.731014 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:29.732157 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:29.733550 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:29.734119 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:29.734314 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:29.735373 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:29.735379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:29.735396 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:29.735939 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:29.736009 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:29.736039 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:29.736164 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:29.736380 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 12:18:29.736598 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":155,"threads_started":1}
I20260620 12:18:29.741305 13801 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe42b53a56949b59892b9953af30b96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 12:18:29.741675 13801 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:29.741815 13801 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe42b53a56949b59892b9953af30b96, State: Initialized, Role: FOLLOWER
I20260620 12:18:29.741889 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:29.742363 13801 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe42b53a56949b59892b9953af30b96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 12:18:29.742548 13801 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:29.742616 13801 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:29.742725 13801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:29.744122 13801 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe42b53a56949b59892b9953af30b96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 12:18:29.744557 13801 leader_election.cc:302] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfe42b53a56949b59892b9953af30b96; no voters:
I20260620 12:18:29.744972 13801 leader_election.cc:288] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:29.745116 13811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:29.745381 13811 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [term 1 LEADER]: Becoming Leader. State: Replica: bfe42b53a56949b59892b9953af30b96, State: Running, Role: LEADER
I20260620 12:18:29.745790 13811 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe42b53a56949b59892b9953af30b96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 12:18:29.746507 13801 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:29.747107 13811 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfe42b53a56949b59892b9953af30b96. Latest consensus state: current_term: 1 leader_uuid: "bfe42b53a56949b59892b9953af30b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe42b53a56949b59892b9953af30b96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } } }
I20260620 12:18:29.747957 13811 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:29.747830 13812 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfe42b53a56949b59892b9953af30b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe42b53a56949b59892b9953af30b96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } } }
I20260620 12:18:29.748332 13812 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:29.748678 13820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:29.749749 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:29.744573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:57661 (local address 127.0.0.1:40839)
0620 12:18:29.744749 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:29.744755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:29.744989 (+ 234us) server_negotiation.cc:407] Connection header received
0620 12:18:29.745185 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:29.745189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:29.745249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:29.745327 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:29.745719 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:29.746344 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:29.747425 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:29.747739 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:29.747813 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:29.749031 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:29.749035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:29.749037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:29.749092 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:29.749124 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:29.749141 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:29.749211 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:29.749555 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 12:18:29.749600 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":67}
I20260620 12:18:29.752673 13820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:29.760232 13820 catalog_manager.cc:1347] Generated new cluster ID: ed3b7f8fa4e345f8ab8ad0fa6d8a9826
I20260620 12:18:29.760354 13820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:29.787225 13820 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:29.788594 13820 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:29.800060 13820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfe42b53a56949b59892b9953af30b96: Generated new TSK 0
I20260620 12:18:29.801411 13820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:29.808684 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13733
W20260620 12:18:29.815384 13718 connection.cc:535] client connection to 127.0.0.1:40839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40839): Connection reset by peer (error 104)
2026-06-20T12:18:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:29 dist-test-slave-rh5v krb5kdc[13832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:29 dist-test-slave-rh5v krb5kdc[13832](info): set up 2 sockets
Jun 20 12:18:29 dist-test-slave-rh5v krb5kdc[13832](info): commencing operation
krb5kdc: starting...
W20260620 12:18:30.785835 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:30 dist-test-slave-rh5v krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:30.920267 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:31.053761 13845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:31.054225 13845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:31.054298 13845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:31.058905 13845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:31.059024 13845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:31.059056 13845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:31.059079 13845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:31.059103 13845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:31.065568 13845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:31.067199 13845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:31.068609 13845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:31.079468 13851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:31.079530 13850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:31.079543 13853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:31.079833 13845 server_base.cc:1034] running on GCE node
I20260620 12:18:31.080683 13845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:31.082237 13845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:31.083422 13845 hybrid_clock.cc:648] HybridClock initialized: now 1781957911083405 us; error 49 us; skew 500 ppm
Jun 20 12:18:31 dist-test-slave-rh5v krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:31.086956 13845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:31.088193 13845 webserver.cc:460] Webserver started at http://127.0.0.1:40829/ using document root <none> and password file <none>
I20260620 12:18:31.088930 13845 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:31.089010 13845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:31.089291 13845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:31.091909 13845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "7d39605805d9457fa0cb84d130995b37"
format_stamp: "Formatted at 2026-06-20 12:18:31 on dist-test-slave-rh5v"
server_key: "03d3b4b1a82549c37887dd5a20236e82"
server_key_iv: "eeab21ce798cd79a33886045363e58e7"
server_key_version: "encryptionkey@0"
I20260620 12:18:31.092573 13845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "7d39605805d9457fa0cb84d130995b37"
format_stamp: "Formatted at 2026-06-20 12:18:31 on dist-test-slave-rh5v"
server_key: "03d3b4b1a82549c37887dd5a20236e82"
server_key_iv: "eeab21ce798cd79a33886045363e58e7"
server_key_version: "encryptionkey@0"
I20260620 12:18:31.097626 13845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:18:31.100992 13860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:31.102746 13845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:18:31.102926 13845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "7d39605805d9457fa0cb84d130995b37"
format_stamp: "Formatted at 2026-06-20 12:18:31 on dist-test-slave-rh5v"
server_key: "03d3b4b1a82549c37887dd5a20236e82"
server_key_iv: "eeab21ce798cd79a33886045363e58e7"
server_key_version: "encryptionkey@0"
I20260620 12:18:31.103060 13845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:31.108599 13845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:31.112274 13845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:31.112493 13845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:31.121274 13845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43821
I20260620 12:18:31.121322 13912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43821 every 8 connection(s)
I20260620 12:18:31.122501 13845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:31.125939 13913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:31.129433 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13845
I20260620 12:18:31.129619 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:31.129946 25140 external_mini_cluster.cc:1428] Setting key 29f99e9b820f63e952adf7700a0944a8
I20260620 12:18:31.133033 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37: Bootstrap starting.
I20260620 12:18:31.135831 13913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:31.136719 13913 log.cc:824] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:31.136814 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:31.131820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:34601 (local address 127.0.0.1:43821)
0620 12:18:31.132423 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:31.132452 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:31.132503 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:18:31.133493 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:31.133518 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:31.133936 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:31.134357 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:31.135251 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:31.135257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:31.135271 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:31.135283 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:31.135375 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:31.135401 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:31.135514 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:31.135627 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 12:18:31.135826 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":192,"threads_started":1}
I20260620 12:18:31.139608 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37: No bootstrap required, opened a new log
I20260620 12:18:31.140808 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:31.144155 13913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39605805d9457fa0cb84d130995b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260620 12:18:31.144562 13913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:31.144635 13913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d39605805d9457fa0cb84d130995b37, State: Initialized, Role: FOLLOWER
I20260620 12:18:31.145272 13913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39605805d9457fa0cb84d130995b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260620 12:18:31.145474 13913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:31.145536 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:31.143405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:41919 (local address 127.0.0.1:43821)
0620 12:18:31.143556 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:31.143560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:31.143769 (+ 209us) server_negotiation.cc:407] Connection header received
0620 12:18:31.143953 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:31.143956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:31.143999 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:31.144064 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:31.145039 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:31.145042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:31.145044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:31.145055 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:31.145136 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:31.145150 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:31.145229 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:31.145342 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 12:18:31.145383 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:18:31.145555 13913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:31.145761 13913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:31.146888 13913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39605805d9457fa0cb84d130995b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260620 12:18:31.147317 13913 leader_election.cc:302] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d39605805d9457fa0cb84d130995b37; no voters:
I20260620 12:18:31.147689 13913 leader_election.cc:288] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:31.147821 13923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:31.148097 13923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [term 1 LEADER]: Becoming Leader. State: Replica: 7d39605805d9457fa0cb84d130995b37, State: Running, Role: LEADER
I20260620 12:18:31.148500 13923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39605805d9457fa0cb84d130995b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260620 12:18:31.149246 13913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:31.150903 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d39605805d9457fa0cb84d130995b37. Latest consensus state: current_term: 1 leader_uuid: "7d39605805d9457fa0cb84d130995b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39605805d9457fa0cb84d130995b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } } }
I20260620 12:18:31.150955 13924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d39605805d9457fa0cb84d130995b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39605805d9457fa0cb84d130995b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } } }
I20260620 12:18:31.151131 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:31.151130 13924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:31.151602 13935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:31.155678 13935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:31.163465 13935 catalog_manager.cc:1347] Generated new cluster ID: bdf50ec4d5934c97bb8806743d0b1a29
I20260620 12:18:31.163581 13935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:31.189083 13935 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:31.189497 13935 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:31.195938 13935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d39605805d9457fa0cb84d130995b37: Generated new TSK 0
I20260620 12:18:31.196851 13935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:31.206923 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13845
2026-06-20T12:18:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:31 dist-test-slave-rh5v krb5kdc[13944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:31 dist-test-slave-rh5v krb5kdc[13944](info): set up 2 sockets
Jun 20 12:18:31 dist-test-slave-rh5v krb5kdc[13944](info): commencing operation
krb5kdc: starting...
W20260620 12:18:32.173911 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:32 dist-test-slave-rh5v krb5kdc[13944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:32.316062 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:32.448151 13957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:32.448529 13957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:32.448575 13957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:32.453229 13957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:32.453333 13957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:32.453363 13957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:32.453385 13957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:32.453405 13957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:32.459823 13957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:32.461421 13957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:32.462751 13957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:32.473557 13963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:32.473562 13965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:32.473773 13962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:32.474359 13957 server_base.cc:1034] running on GCE node
I20260620 12:18:32.475113 13957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:32.476572 13957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:32.477813 13957 hybrid_clock.cc:648] HybridClock initialized: now 1781957912477774 us; error 40 us; skew 500 ppm
Jun 20 12:18:32 dist-test-slave-rh5v krb5kdc[13944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:32.485139 13957 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:32.486581 13957 webserver.cc:460] Webserver started at http://127.0.0.1:35083/ using document root <none> and password file <none>
I20260620 12:18:32.487586 13957 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:32.487708 13957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:32.488034 13957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:32.491750 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "1059f654870b4294bed4a124a157d27d"
format_stamp: "Formatted at 2026-06-20 12:18:32 on dist-test-slave-rh5v"
server_key: "8185cf087852a37c7091b6a915ffa2c1"
server_key_iv: "ff3624215ccf900e66bae07be8c0ce73"
server_key_version: "encryptionkey@0"
I20260620 12:18:32.492769 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "1059f654870b4294bed4a124a157d27d"
format_stamp: "Formatted at 2026-06-20 12:18:32 on dist-test-slave-rh5v"
server_key: "8185cf087852a37c7091b6a915ffa2c1"
server_key_iv: "ff3624215ccf900e66bae07be8c0ce73"
server_key_version: "encryptionkey@0"
I20260620 12:18:32.499747 13957 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 12:18:32.504082 13972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:32.506448 13957 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 12:18:32.506670 13957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "1059f654870b4294bed4a124a157d27d"
format_stamp: "Formatted at 2026-06-20 12:18:32 on dist-test-slave-rh5v"
server_key: "8185cf087852a37c7091b6a915ffa2c1"
server_key_iv: "ff3624215ccf900e66bae07be8c0ce73"
server_key_version: "encryptionkey@0"
I20260620 12:18:32.506856 13957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:32.546774 13957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:32.550683 13957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:32.550881 13957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:32.559641 13957 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44799
I20260620 12:18:32.559767 14024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44799 every 8 connection(s)
I20260620 12:18:32.560963 13957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:32.564841 14025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:32.565613 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 13957
I20260620 12:18:32.565766 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:32.566141 25140 external_mini_cluster.cc:1428] Setting key abafe522527889565abb9c833fd588eb
I20260620 12:18:32.573382 14025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d: Bootstrap starting.
I20260620 12:18:32.576376 14025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:32.577590 14025 log.cc:824] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:18:32 dist-test-slave-rh5v krb5kdc[13944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:18:32.580894 14025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d: No bootstrap required, opened a new log
I20260620 12:18:32.586361 14025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1059f654870b4294bed4a124a157d27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260620 12:18:32.586601 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:32.568232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:44867 (local address 127.0.0.1:44799)
0620 12:18:32.568967 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:32.568992 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:32.569178 (+ 186us) server_negotiation.cc:407] Connection header received
0620 12:18:32.570228 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:32.570258 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:32.570809 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:32.571274 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:32.572199 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:32.573778 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:32.574888 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:32.575639 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:32.575850 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:32.579189 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:32.579197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:32.579219 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:32.579262 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:32.582053 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:32.582952 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:32.582961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:32.582969 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:32.583186 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:32.583678 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:32.583683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:32.583685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:32.584416 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:32.584665 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:32.585165 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 12:18:32.585477 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":203,"threads_started":1}
I20260620 12:18:32.586707 14025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:32.586757 14025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1059f654870b4294bed4a124a157d27d, State: Initialized, Role: FOLLOWER
I20260620 12:18:32.587270 14025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1059f654870b4294bed4a124a157d27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260620 12:18:32.587471 14025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:32.587618 14025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:32.587738 14025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:32.589861 14025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1059f654870b4294bed4a124a157d27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260620 12:18:32.590292 14025 leader_election.cc:302] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1059f654870b4294bed4a124a157d27d; no voters:
I20260620 12:18:32.590672 14025 leader_election.cc:288] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:32.590809 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:32.590890 14030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:32.591213 14030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [term 1 LEADER]: Becoming Leader. State: Replica: 1059f654870b4294bed4a124a157d27d, State: Running, Role: LEADER
I20260620 12:18:32.591712 14030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1059f654870b4294bed4a124a157d27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260620 12:18:32.592166 14025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:32.593751 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1059f654870b4294bed4a124a157d27d. Latest consensus state: current_term: 1 leader_uuid: "1059f654870b4294bed4a124a157d27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1059f654870b4294bed4a124a157d27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } } }
I20260620 12:18:32.593972 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:32.594190 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1059f654870b4294bed4a124a157d27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1059f654870b4294bed4a124a157d27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } } }
I20260620 12:18:32.594380 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d [sys.catalog]: This master's current role is: LEADER
W20260620 12:18:32.598018 14050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:32.598129 14050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:32.598222 14046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:32.599401 14046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:32.607642 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:32.593957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:58693 (local address 127.0.0.1:44799)
0620 12:18:32.594106 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:32.594112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:32.595429 (+ 1317us) server_negotiation.cc:407] Connection header received
0620 12:18:32.595584 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:32.595587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:32.595654 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:32.595782 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:32.596776 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:32.597610 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:32.600288 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:32.600631 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:32.600743 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:32.602425 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:32.602431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:32.602435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:32.602484 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:32.604479 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:32.605504 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:32.605509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:32.605510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:32.605662 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:32.606224 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:32.606228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:32.606230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:32.606509 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:32.606688 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:32.607316 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 12:18:32.607467 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:18:32.608222 14046 catalog_manager.cc:1347] Generated new cluster ID: 5ff4455840d5432198f3e3520f4ed774
I20260620 12:18:32.608325 14046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:32.618157 14046 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:32.619854 14046 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:32.627705 14046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1059f654870b4294bed4a124a157d27d: Generated new TSK 0
I20260620 12:18:32.628962 14046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:32.642716 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 13957
W20260620 12:18:32.649809 13942 connection.cc:535] client connection to 127.0.0.1:44799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44799): Connection reset by peer (error 104)
2026-06-20T12:18:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:32 dist-test-slave-rh5v krb5kdc[14056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:32 dist-test-slave-rh5v krb5kdc[14056](info): set up 2 sockets
Jun 20 12:18:32 dist-test-slave-rh5v krb5kdc[14056](info): commencing operation
krb5kdc: starting...
W20260620 12:18:33.618194 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:33 dist-test-slave-rh5v krb5kdc[14056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:33.753829 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:33.890491 14069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:33.890865 14069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:33.890913 14069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:33.895452 14069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:33.895542 14069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:33.895561 14069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:33.895579 14069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:33.895593 14069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:33.901800 14069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:33.903290 14069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:33.904652 14069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:33.915463 14077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:33.915469 14074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:33.915452 14075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:33.915925 14069 server_base.cc:1034] running on GCE node
I20260620 12:18:33.916643 14069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:33.918045 14069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:33.919283 14069 hybrid_clock.cc:648] HybridClock initialized: now 1781957913919243 us; error 58 us; skew 500 ppm
Jun 20 12:18:33 dist-test-slave-rh5v krb5kdc[14056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:33.922890 14069 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:33.924062 14069 webserver.cc:460] Webserver started at http://127.0.0.1:41395/ using document root <none> and password file <none>
I20260620 12:18:33.924721 14069 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:33.924803 14069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:33.925009 14069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:33.927431 14069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "ba34a9ea2ac843719d7aee224f6fe377"
format_stamp: "Formatted at 2026-06-20 12:18:33 on dist-test-slave-rh5v"
server_key: "c2f7ab5801e23f732e482dd0da9643ff"
server_key_iv: "684788f97f57df08725cd23f13a69ac3"
server_key_version: "encryptionkey@0"
I20260620 12:18:33.928081 14069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "ba34a9ea2ac843719d7aee224f6fe377"
format_stamp: "Formatted at 2026-06-20 12:18:33 on dist-test-slave-rh5v"
server_key: "c2f7ab5801e23f732e482dd0da9643ff"
server_key_iv: "684788f97f57df08725cd23f13a69ac3"
server_key_version: "encryptionkey@0"
I20260620 12:18:33.932925 14069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 12:18:33.936385 14084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:33.938319 14069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:33.938531 14069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "ba34a9ea2ac843719d7aee224f6fe377"
format_stamp: "Formatted at 2026-06-20 12:18:33 on dist-test-slave-rh5v"
server_key: "c2f7ab5801e23f732e482dd0da9643ff"
server_key_iv: "684788f97f57df08725cd23f13a69ac3"
server_key_version: "encryptionkey@0"
I20260620 12:18:33.938685 14069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:33.954916 14069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:33.958750 14069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:33.958964 14069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:33.966840 14069 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43379
I20260620 12:18:33.967027 14136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43379 every 8 connection(s)
I20260620 12:18:33.967990 14069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:33.972034 14137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:33.972734 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14069
I20260620 12:18:33.972884 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:33.973248 25140 external_mini_cluster.cc:1428] Setting key e8dd81722bc81559046207faf0bc69d5
I20260620 12:18:33.980532 14137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377: Bootstrap starting.
I20260620 12:18:33.984011 14137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:33.985116 14137 log.cc:824] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:33.985682 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:33.975125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:39007 (local address 127.0.0.1:43379)
0620 12:18:33.975764 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:33.975787 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:33.975849 (+ 62us) server_negotiation.cc:407] Connection header received
0620 12:18:33.976811 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:33.976838 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:33.977448 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:33.977919 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:33.978429 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:33.979987 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:33.981168 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:33.981904 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:33.982120 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:33.983150 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:33.983157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:33.983177 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:33.983766 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:33.983859 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:33.983895 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:33.984050 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:33.984321 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 12:18:33.984508 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":157,"threads_started":1}
I20260620 12:18:33.988080 14137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377: No bootstrap required, opened a new log
I20260620 12:18:33.989990 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:33.992693 14137 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba34a9ea2ac843719d7aee224f6fe377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260620 12:18:33.993084 14137 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:33.993165 14137 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba34a9ea2ac843719d7aee224f6fe377, State: Initialized, Role: FOLLOWER
I20260620 12:18:33.993775 14137 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba34a9ea2ac843719d7aee224f6fe377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260620 12:18:33.993963 14137 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:33.994042 14137 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:33.994153 14137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:33.995456 14137 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba34a9ea2ac843719d7aee224f6fe377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260620 12:18:33.995887 14137 leader_election.cc:302] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba34a9ea2ac843719d7aee224f6fe377; no voters:
I20260620 12:18:33.996320 14137 leader_election.cc:288] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:33.996461 14147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:33.996655 14147 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [term 1 LEADER]: Becoming Leader. State: Replica: ba34a9ea2ac843719d7aee224f6fe377, State: Running, Role: LEADER
I20260620 12:18:33.997011 14147 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba34a9ea2ac843719d7aee224f6fe377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260620 12:18:33.997330 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:33.992375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:45447 (local address 127.0.0.1:43379)
0620 12:18:33.992526 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:33.992531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:33.992552 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:18:33.992702 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:33.992705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:33.992767 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:33.992882 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:33.993264 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:33.994093 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:33.995021 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:33.995384 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:33.995517 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:33.996763 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:33.996768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:33.996771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:33.996787 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:33.996820 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:33.996834 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:33.996895 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:33.997135 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 12:18:33.997178 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:18:33.998059 14137 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:33.999267 14147 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba34a9ea2ac843719d7aee224f6fe377. Latest consensus state: current_term: 1 leader_uuid: "ba34a9ea2ac843719d7aee224f6fe377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba34a9ea2ac843719d7aee224f6fe377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } } }
I20260620 12:18:33.999410 14147 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:33.999802 14148 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba34a9ea2ac843719d7aee224f6fe377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba34a9ea2ac843719d7aee224f6fe377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } } }
I20260620 12:18:34.000043 14156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:34.000072 14148 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:34.003955 14156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:34.012732 14156 catalog_manager.cc:1347] Generated new cluster ID: 9277a1b9d0074ef09e1de9d239db8b35
I20260620 12:18:34.012844 14156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:34.029136 14156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:34.030431 14156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:34.044867 14156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba34a9ea2ac843719d7aee224f6fe377: Generated new TSK 0
I20260620 12:18:34.045980 14156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:34.058282 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14069
W20260620 12:18:34.064388 14054 connection.cc:535] client connection to 127.0.0.1:43379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43379): Connection reset by peer (error 104)
2026-06-20T12:18:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:34 dist-test-slave-rh5v krb5kdc[14168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:34 dist-test-slave-rh5v krb5kdc[14168](info): set up 2 sockets
Jun 20 12:18:34 dist-test-slave-rh5v krb5kdc[14168](info): commencing operation
krb5kdc: starting...
W20260620 12:18:35.029693 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:35 dist-test-slave-rh5v krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:18:35.167905 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:35.305601 14181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:35.305969 14181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:35.306015 14181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:35.310523 14181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:35.310627 14181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:35.310647 14181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:35.310664 14181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:35.310679 14181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:35.316784 14181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:35.318286 14181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:35.319602 14181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:35.330061 14187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:35.330062 14186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:35.330245 14181 server_base.cc:1034] running on GCE node
W20260620 12:18:35.330071 14189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:35.331107 14181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:35.332510 14181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:35.333760 14181 hybrid_clock.cc:648] HybridClock initialized: now 1781957915333734 us; error 47 us; skew 500 ppm
Jun 20 12:18:35 dist-test-slave-rh5v krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:35.340159 14181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:18:35.341454 14181 webserver.cc:460] Webserver started at http://127.0.0.1:44605/ using document root <none> and password file <none>
I20260620 12:18:35.342123 14181 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:35.342209 14181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:35.342465 14181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:35.344935 14181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "0e8d6172f89d47d7b6bb87b833dbf476"
format_stamp: "Formatted at 2026-06-20 12:18:35 on dist-test-slave-rh5v"
server_key: "b8e1aee7c616405d6348244171e6ed9f"
server_key_iv: "fe66a57e4ef708d5049bbd3d30fbc564"
server_key_version: "encryptionkey@0"
I20260620 12:18:35.345676 14181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "0e8d6172f89d47d7b6bb87b833dbf476"
format_stamp: "Formatted at 2026-06-20 12:18:35 on dist-test-slave-rh5v"
server_key: "b8e1aee7c616405d6348244171e6ed9f"
server_key_iv: "fe66a57e4ef708d5049bbd3d30fbc564"
server_key_version: "encryptionkey@0"
I20260620 12:18:35.350472 14181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:35.354379 14196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:35.356145 14181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:18:35.356353 14181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "0e8d6172f89d47d7b6bb87b833dbf476"
format_stamp: "Formatted at 2026-06-20 12:18:35 on dist-test-slave-rh5v"
server_key: "b8e1aee7c616405d6348244171e6ed9f"
server_key_iv: "fe66a57e4ef708d5049bbd3d30fbc564"
server_key_version: "encryptionkey@0"
I20260620 12:18:35.356528 14181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:35.362671 14181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:35.366421 14181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:35.366652 14181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:35.375811 14181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33195
I20260620 12:18:35.375908 14248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33195 every 8 connection(s)
I20260620 12:18:35.377238 14181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:35.381409 14249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:35.387333 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14181
I20260620 12:18:35.387535 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:35.387884 25140 external_mini_cluster.cc:1428] Setting key 92cb84cdec3c6a7749620e6b5bccc7b5
I20260620 12:18:35.389206 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476: Bootstrap starting.
I20260620 12:18:35.392297 14249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:35.393462 14249 log.cc:824] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:35.395449 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:35.390031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:44533 (local address 127.0.0.1:33195)
0620 12:18:35.390624 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:35.390641 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:35.390695 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:18:35.391523 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:35.391545 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:35.391950 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:35.392324 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:35.393435 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:35.393442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:35.393458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:35.393472 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:35.393552 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:35.393586 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:35.393732 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:35.394116 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 12:18:35.394343 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":167,"threads_started":1}
I20260620 12:18:35.396355 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476: No bootstrap required, opened a new log
I20260620 12:18:35.399431 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:35.401114 14249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260620 12:18:35.401433 14249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:35.401473 14249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8d6172f89d47d7b6bb87b833dbf476, State: Initialized, Role: FOLLOWER
I20260620 12:18:35.402063 14249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260620 12:18:35.402273 14249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:35.402360 14249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:35.402482 14249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:35.403916 14249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260620 12:18:35.404415 14249 leader_election.cc:302] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e8d6172f89d47d7b6bb87b833dbf476; no voters:
I20260620 12:18:35.404539 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:35.402147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:55255 (local address 127.0.0.1:33195)
0620 12:18:35.402369 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:35.402375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:35.402655 (+ 280us) server_negotiation.cc:407] Connection header received
0620 12:18:35.402930 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:35.402934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:35.402992 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:35.403106 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:35.404017 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:35.404022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:35.404025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:35.404040 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:35.404079 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:35.404093 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:35.404209 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:35.404313 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 12:18:35.404365 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:18:35.404872 14249 leader_election.cc:288] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:35.405105 14259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:35.405407 14259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [term 1 LEADER]: Becoming Leader. State: Replica: 0e8d6172f89d47d7b6bb87b833dbf476, State: Running, Role: LEADER
I20260620 12:18:35.405860 14259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260620 12:18:35.406610 14249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:35.407547 14259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8d6172f89d47d7b6bb87b833dbf476. Latest consensus state: current_term: 1 leader_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } } }
I20260620 12:18:35.407729 14259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:35.408016 14260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8d6172f89d47d7b6bb87b833dbf476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } } }
I20260620 12:18:35.408197 14260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:35.408207 14268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:35.412403 14268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:35.420670 14268 catalog_manager.cc:1347] Generated new cluster ID: 5de2b4968d1f4aa48b5c621d6960108e
I20260620 12:18:35.420796 14268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:35.443358 14268 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:35.443816 14268 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:35.450959 14268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e8d6172f89d47d7b6bb87b833dbf476: Generated new TSK 0
I20260620 12:18:35.451931 14268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:35.466616 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14181
2026-06-20T12:18:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:35 dist-test-slave-rh5v krb5kdc[14280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:35 dist-test-slave-rh5v krb5kdc[14280](info): set up 2 sockets
Jun 20 12:18:35 dist-test-slave-rh5v krb5kdc[14280](info): commencing operation
krb5kdc: starting...
W20260620 12:18:36.441833 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:36 dist-test-slave-rh5v krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:36.580125 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40383
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:36.711232 14293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:36.711606 14293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:36.711653 14293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:36.716048 14293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:36.716153 14293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:36.716171 14293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:36.716190 14293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:36.716205 14293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:36.722467 14293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:40383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:36.724012 14293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:36.725418 14293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:36.735894 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:36.735925 14301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:36.736384 14293 server_base.cc:1034] running on GCE node
W20260620 12:18:36.735894 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:36.737198 14293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:36.738700 14293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:36.739888 14293 hybrid_clock.cc:648] HybridClock initialized: now 1781957916739869 us; error 50 us; skew 500 ppm
Jun 20 12:18:36 dist-test-slave-rh5v krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957916, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:36.743306 14293 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:36.744517 14293 webserver.cc:460] Webserver started at http://127.24.141.62:32869/ using document root <none> and password file <none>
I20260620 12:18:36.745223 14293 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:36.745299 14293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:36.745530 14293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:36.747963 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "43675794398441d2aa457f156a8be9ce"
format_stamp: "Formatted at 2026-06-20 12:18:36 on dist-test-slave-rh5v"
server_key: "06c1191d8716a747783445c092e3d5e9"
server_key_iv: "1995bf1434caad69e98fd035dcf23fce"
server_key_version: "encryptionkey@0"
I20260620 12:18:36.748585 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "43675794398441d2aa457f156a8be9ce"
format_stamp: "Formatted at 2026-06-20 12:18:36 on dist-test-slave-rh5v"
server_key: "06c1191d8716a747783445c092e3d5e9"
server_key_iv: "1995bf1434caad69e98fd035dcf23fce"
server_key_version: "encryptionkey@0"
I20260620 12:18:36.753388 14293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:18:36.756844 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:36.758771 14293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:36.758951 14293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "43675794398441d2aa457f156a8be9ce"
format_stamp: "Formatted at 2026-06-20 12:18:36 on dist-test-slave-rh5v"
server_key: "06c1191d8716a747783445c092e3d5e9"
server_key_iv: "1995bf1434caad69e98fd035dcf23fce"
server_key_version: "encryptionkey@0"
I20260620 12:18:36.759117 14293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:36.779347 14293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:36.783768 14293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:36.784014 14293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:36.794773 14293 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:40383
I20260620 12:18:36.794806 14360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:40383 every 8 connection(s)
I20260620 12:18:36.796121 14293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:36.799341 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14293
I20260620 12:18:36.799523 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:36.799855 25140 external_mini_cluster.cc:1428] Setting key 2ceb3337ad3c8d6d521e6feab8c9ffc3
I20260620 12:18:36.799943 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:36.808506 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce: Bootstrap starting.
Jun 20 12:18:36 dist-test-slave-rh5v krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:36.811539 14361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:36.812414 14361 log.cc:824] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:36.815212 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce: No bootstrap required, opened a new log
I20260620 12:18:36.818164 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:36.801836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34501 (local address 127.24.141.62:40383)
0620 12:18:36.802483 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:36.802506 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:36.802560 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:18:36.803836 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:36.803868 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:36.804409 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:36.804874 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:36.805954 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:36.807187 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:36.808157 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:36.808804 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:36.808975 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:36.812224 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:36.812231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:36.812252 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:36.812292 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:36.814388 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:36.815160 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:36.815166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:36.815172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:36.815312 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:36.815707 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:36.815710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:36.815711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:36.816245 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:36.816548 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:36.816815 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 12:18:36.817149 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":189,"threads_started":1}
I20260620 12:18:36.819422 14361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43675794398441d2aa457f156a8be9ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40383 } }
I20260620 12:18:36.819751 14361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:36.819818 14361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43675794398441d2aa457f156a8be9ce, State: Initialized, Role: FOLLOWER
I20260620 12:18:36.820315 14361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43675794398441d2aa457f156a8be9ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40383 } }
I20260620 12:18:36.820468 14361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:36.820551 14361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:36.820649 14361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:36.822031 14361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43675794398441d2aa457f156a8be9ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40383 } }
I20260620 12:18:36.822347 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:36.822507 14361 leader_election.cc:302] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43675794398441d2aa457f156a8be9ce; no voters:
I20260620 12:18:36.822942 14361 leader_election.cc:288] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:36.823076 14366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:36.823273 14366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [term 1 LEADER]: Becoming Leader. State: Replica: 43675794398441d2aa457f156a8be9ce, State: Running, Role: LEADER
I20260620 12:18:36.823765 14366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43675794398441d2aa457f156a8be9ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40383 } }
I20260620 12:18:36.824573 14361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:36.825711 14368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43675794398441d2aa457f156a8be9ce. Latest consensus state: current_term: 1 leader_uuid: "43675794398441d2aa457f156a8be9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43675794398441d2aa457f156a8be9ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40383 } } }
I20260620 12:18:36.825891 14368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:36.826387 14380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:36.826395 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43675794398441d2aa457f156a8be9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43675794398441d2aa457f156a8be9ce" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 40383 } } }
I20260620 12:18:36.826612 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:36.831175 14380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:36.838745 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:36.825689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34041 (local address 127.24.141.62:40383)
0620 12:18:36.825831 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:36.825836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:36.827473 (+ 1637us) server_negotiation.cc:407] Connection header received
0620 12:18:36.827662 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:36.827665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:36.827738 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:36.827881 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:36.828942 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:36.829868 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:36.831139 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:36.831457 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:36.831563 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:36.833621 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:36.833625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:36.833627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:36.833662 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:36.835791 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:36.836756 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:36.836760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:36.836762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:36.836938 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:36.837495 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:36.837499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:36.837501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:36.837794 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:36.837966 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:36.838416 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 12:18:36.838536 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:18:36.841312 14380 catalog_manager.cc:1347] Generated new cluster ID: ab8f39de986340b5969260f579c0067d
I20260620 12:18:36.841435 14380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:36.865161 14380 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:36.867215 14380 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:36.879185 14380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43675794398441d2aa457f156a8be9ce: Generated new TSK 0
I20260620 12:18:36.880599 14380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:36.894064 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14293
W20260620 12:18:36.901834 14278 connection.cc:535] client connection to 127.24.141.62:40383 recv error: Network error: failed to read from TLS socket (remote: 127.24.141.62:40383): Connection reset by peer (error 104)
2026-06-20T12:18:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:36 dist-test-slave-rh5v krb5kdc[14392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:36 dist-test-slave-rh5v krb5kdc[14392](info): set up 2 sockets
Jun 20 12:18:36 dist-test-slave-rh5v krb5kdc[14392](info): commencing operation
krb5kdc: starting...
W20260620 12:18:37.873885 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:37 dist-test-slave-rh5v krb5kdc[14392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:38.013167 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38315
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:38315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:38.147168 14405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:38.147585 14405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:38.147661 14405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:38.152318 14405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:38.152431 14405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:38.152451 14405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:38.152468 14405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:38.152483 14405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:38.159363 14405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:38315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:38.161123 14405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:38.162585 14405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:18:38.173990 14405 server_base.cc:1034] running on GCE node
W20260620 12:18:38.173921 14413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:38.173921 14411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:38.173921 14410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:38.175267 14405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:38.176889 14405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:38.178104 14405 hybrid_clock.cc:648] HybridClock initialized: now 1781957918178085 us; error 47 us; skew 500 ppm
Jun 20 12:18:38 dist-test-slave-rh5v krb5kdc[14392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957918, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:38.181533 14405 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:38.182708 14405 webserver.cc:460] Webserver started at http://127.24.141.62:43927/ using document root <none> and password file <none>
I20260620 12:18:38.183413 14405 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:38.183488 14405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:38.183727 14405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:38.186430 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "84da522e2f694f058e6fb6e21cb6ba45"
format_stamp: "Formatted at 2026-06-20 12:18:38 on dist-test-slave-rh5v"
server_key: "a2002ce26684c9f5f17aa4b6b3d65a57"
server_key_iv: "f98aebd34be107c385676b35da10a7c4"
server_key_version: "encryptionkey@0"
I20260620 12:18:38.187110 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "84da522e2f694f058e6fb6e21cb6ba45"
format_stamp: "Formatted at 2026-06-20 12:18:38 on dist-test-slave-rh5v"
server_key: "a2002ce26684c9f5f17aa4b6b3d65a57"
server_key_iv: "f98aebd34be107c385676b35da10a7c4"
server_key_version: "encryptionkey@0"
I20260620 12:18:38.192462 14405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:38.196089 14420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:38.198017 14405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:38.198213 14405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "84da522e2f694f058e6fb6e21cb6ba45"
format_stamp: "Formatted at 2026-06-20 12:18:38 on dist-test-slave-rh5v"
server_key: "a2002ce26684c9f5f17aa4b6b3d65a57"
server_key_iv: "f98aebd34be107c385676b35da10a7c4"
server_key_version: "encryptionkey@0"
I20260620 12:18:38.198375 14405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:38.230316 14405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:38.234684 14405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:38.234912 14405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:38.243891 14405 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:38315
I20260620 12:18:38.243937 14472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:38315 every 8 connection(s)
I20260620 12:18:38.245136 14405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:38.248960 14473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:38.253094 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14405
I20260620 12:18:38.253278 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:38.253621 25140 external_mini_cluster.cc:1428] Setting key 882a06c84caee3dfdb508e9c99fc707d
I20260620 12:18:38.257464 14473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45: Bootstrap starting.
I20260620 12:18:38.260643 14473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:38.261600 14473 log.cc:824] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:38.264899 14473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45: No bootstrap required, opened a new log
I20260620 12:18:38.266605 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:38.255804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48247 (local address 127.24.141.62:38315)
0620 12:18:38.256499 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:38.256524 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:38.256586 (+ 62us) server_negotiation.cc:407] Connection header received
0620 12:18:38.257681 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:38.257711 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:38.258254 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:38.258686 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:38.259215 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:38.260724 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:38.261946 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:38.262597 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:38.262772 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:38.263962 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:38.263970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:38.263991 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:38.264592 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:38.264690 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:38.264731 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:38.264915 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:38.265247 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 12:18:38.265464 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":182,"threads_started":1}
I20260620 12:18:38.270231 14473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84da522e2f694f058e6fb6e21cb6ba45" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38315 } }
I20260620 12:18:38.270649 14473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:38.270742 14473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84da522e2f694f058e6fb6e21cb6ba45, State: Initialized, Role: FOLLOWER
I20260620 12:18:38.271317 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:38.271317 14473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84da522e2f694f058e6fb6e21cb6ba45" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38315 } }
I20260620 12:18:38.271533 14473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:38.271615 14473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:38.271735 14473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:38.273353 14473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84da522e2f694f058e6fb6e21cb6ba45" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38315 } }
I20260620 12:18:38.273895 14473 leader_election.cc:302] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84da522e2f694f058e6fb6e21cb6ba45; no voters:
I20260620 12:18:38.274350 14473 leader_election.cc:288] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:38.274461 14482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:38.274845 14482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [term 1 LEADER]: Becoming Leader. State: Replica: 84da522e2f694f058e6fb6e21cb6ba45, State: Running, Role: LEADER
I20260620 12:18:38.275256 14482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84da522e2f694f058e6fb6e21cb6ba45" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38315 } }
I20260620 12:18:38.276060 14473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:38.276696 14482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84da522e2f694f058e6fb6e21cb6ba45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84da522e2f694f058e6fb6e21cb6ba45" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38315 } } }
I20260620 12:18:38.276852 14482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:38.277123 14482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84da522e2f694f058e6fb6e21cb6ba45. Latest consensus state: current_term: 1 leader_uuid: "84da522e2f694f058e6fb6e21cb6ba45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84da522e2f694f058e6fb6e21cb6ba45" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38315 } } }
I20260620 12:18:38.277360 14482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:38.279512 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:38.274426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57043 (local address 127.24.141.62:38315)
0620 12:18:38.274749 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:38.274755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:38.274808 (+ 53us) server_negotiation.cc:407] Connection header received
0620 12:18:38.275071 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:38.275075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:38.275136 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:38.275235 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:38.275551 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:38.276139 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:38.277276 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:38.277609 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:38.277690 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:38.278927 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:38.278931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:38.278935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:38.278954 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:38.278988 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:38.279004 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:38.279064 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:38.279286 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 12:18:38.279330 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
W20260620 12:18:38.282466 14498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:38.282606 14498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:38.282737 14488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:38.283913 14488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:38.291182 14488 catalog_manager.cc:1347] Generated new cluster ID: 9a1505c973bc4d04a09b0b9b5d85945a
I20260620 12:18:38.291285 14488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:38.297676 14488 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:38.299012 14488 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:38.309945 14488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84da522e2f694f058e6fb6e21cb6ba45: Generated new TSK 0
I20260620 12:18:38.311122 14488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:38.316438 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14405
W20260620 12:18:38.323429 14390 connection.cc:535] client connection to 127.24.141.62:38315 recv error: Network error: failed to read from TLS socket (remote: 127.24.141.62:38315): Connection reset by peer (error 104)
2026-06-20T12:18:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:38 dist-test-slave-rh5v krb5kdc[14504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:38 dist-test-slave-rh5v krb5kdc[14504](info): set up 2 sockets
Jun 20 12:18:38 dist-test-slave-rh5v krb5kdc[14504](info): commencing operation
krb5kdc: starting...
W20260620 12:18:39.286021 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:39 dist-test-slave-rh5v krb5kdc[14504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:39.425316 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38341
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:39.561103 14517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:39.561468 14517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:39.561515 14517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:39.565912 14517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:39.566021 14517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:39.566041 14517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:39.566058 14517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:39.566074 14517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:39.572257 14517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:38341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:39.573822 14517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:39.575129 14517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:39.585536 14523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:39.585562 14522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:39.585561 14525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:39.586179 14517 server_base.cc:1034] running on GCE node
I20260620 12:18:39.587036 14517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:39.588583 14517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:39.589766 14517 hybrid_clock.cc:648] HybridClock initialized: now 1781957919589751 us; error 36 us; skew 500 ppm
Jun 20 12:18:39 dist-test-slave-rh5v krb5kdc[14504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957919, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:39.593289 14517 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:39.594676 14517 webserver.cc:460] Webserver started at http://127.24.141.62:36381/ using document root <none> and password file <none>
I20260620 12:18:39.595427 14517 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:39.595507 14517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:39.595788 14517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:39.598496 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "560a1cc6610d4baeb1affae1c8855d8e"
format_stamp: "Formatted at 2026-06-20 12:18:39 on dist-test-slave-rh5v"
server_key: "3eea16d79775d09200c04c3c847b7bf6"
server_key_iv: "d7af3eac03752abcdaa845c21a368006"
server_key_version: "encryptionkey@0"
I20260620 12:18:39.599182 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "560a1cc6610d4baeb1affae1c8855d8e"
format_stamp: "Formatted at 2026-06-20 12:18:39 on dist-test-slave-rh5v"
server_key: "3eea16d79775d09200c04c3c847b7bf6"
server_key_iv: "d7af3eac03752abcdaa845c21a368006"
server_key_version: "encryptionkey@0"
I20260620 12:18:39.604192 14517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:39.607631 14532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:39.609547 14517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:39.609732 14517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "560a1cc6610d4baeb1affae1c8855d8e"
format_stamp: "Formatted at 2026-06-20 12:18:39 on dist-test-slave-rh5v"
server_key: "3eea16d79775d09200c04c3c847b7bf6"
server_key_iv: "d7af3eac03752abcdaa845c21a368006"
server_key_version: "encryptionkey@0"
I20260620 12:18:39.609869 14517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:39.615491 14517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:39.619236 14517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:39.619441 14517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:39.628865 14517 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:38341
I20260620 12:18:39.628942 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:38341 every 8 connection(s)
I20260620 12:18:39.630251 14517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:39.634075 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14517
I20260620 12:18:39.634249 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:39.634618 25140 external_mini_cluster.cc:1428] Setting key 14c03cfdbd5ffab82aea6616ae5151dc
I20260620 12:18:39.634840 14585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:39.642825 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:39.636650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40445 (local address 127.24.141.62:38341)
0620 12:18:39.637400 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:39.637423 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:39.637676 (+ 253us) server_negotiation.cc:407] Connection header received
0620 12:18:39.638918 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:39.638949 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:39.639433 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:39.639942 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:39.641108 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:39.641115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:39.641133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:39.641146 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:39.641219 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:39.641245 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:39.641397 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:39.641505 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 12:18:39.641772 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":213,"threads_started":1}
I20260620 12:18:39.643617 14585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e: Bootstrap starting.
I20260620 12:18:39.646490 14585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:39.646579 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:39.647552 14585 log.cc:824] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:39.650547 14585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e: No bootstrap required, opened a new log
I20260620 12:18:39.651185 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:39.649300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46117 (local address 127.24.141.62:38341)
0620 12:18:39.649475 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:39.649480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:39.649725 (+ 245us) server_negotiation.cc:407] Connection header received
0620 12:18:39.649873 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:39.649876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:39.649922 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:39.649993 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:39.650743 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:39.650747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:39.650750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:39.650763 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:39.650796 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:39.650808 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:39.650889 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:39.650956 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 12:18:39.651006 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:18:39.654968 14585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a1cc6610d4baeb1affae1c8855d8e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38341 } }
I20260620 12:18:39.655316 14585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:39.655378 14585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560a1cc6610d4baeb1affae1c8855d8e, State: Initialized, Role: FOLLOWER
I20260620 12:18:39.655848 14585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a1cc6610d4baeb1affae1c8855d8e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38341 } }
I20260620 12:18:39.656005 14585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:39.656076 14585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:39.656178 14585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:39.657341 14585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a1cc6610d4baeb1affae1c8855d8e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38341 } }
I20260620 12:18:39.657737 14585 leader_election.cc:302] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560a1cc6610d4baeb1affae1c8855d8e; no voters:
I20260620 12:18:39.658104 14585 leader_election.cc:288] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:39.658295 14595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:39.658694 14595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [term 1 LEADER]: Becoming Leader. State: Replica: 560a1cc6610d4baeb1affae1c8855d8e, State: Running, Role: LEADER
I20260620 12:18:39.659128 14595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a1cc6610d4baeb1affae1c8855d8e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38341 } }
I20260620 12:18:39.659561 14585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:39.661405 14596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "560a1cc6610d4baeb1affae1c8855d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a1cc6610d4baeb1affae1c8855d8e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38341 } } }
I20260620 12:18:39.661646 14596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:39.662015 14597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 560a1cc6610d4baeb1affae1c8855d8e. Latest consensus state: current_term: 1 leader_uuid: "560a1cc6610d4baeb1affae1c8855d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a1cc6610d4baeb1affae1c8855d8e" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38341 } } }
I20260620 12:18:39.662417 14597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e [sys.catalog]: This master's current role is: LEADER
W20260620 12:18:39.665540 14610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:39.665670 14610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:39.665874 14609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:39.667179 14609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:39.674871 14609 catalog_manager.cc:1347] Generated new cluster ID: 269075954cb649cc9957f4136a00f997
I20260620 12:18:39.675004 14609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:39.699041 14609 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:39.699585 14609 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:39.707147 14609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 560a1cc6610d4baeb1affae1c8855d8e: Generated new TSK 0
I20260620 12:18:39.708220 14609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:39.721859 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14517
2026-06-20T12:18:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:39 dist-test-slave-rh5v krb5kdc[14616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:39 dist-test-slave-rh5v krb5kdc[14616](info): set up 2 sockets
Jun 20 12:18:39 dist-test-slave-rh5v krb5kdc[14616](info): commencing operation
krb5kdc: starting...
W20260620 12:18:40.697883 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:40 dist-test-slave-rh5v krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:40.836444 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43021
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:43021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:40.970501 14629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:40.970878 14629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:40.970927 14629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:40.975382 14629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:40.975481 14629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:40.975502 14629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:40.975517 14629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:40.975533 14629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:40.981925 14629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:43021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:40.983394 14629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:40.984964 14629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:40.996723 14634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:40.996866 14635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:40.996982 14629 server_base.cc:1034] running on GCE node
W20260620 12:18:40.996723 14637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:40.997975 14629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:40.999682 14629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:41.000913 14629 hybrid_clock.cc:648] HybridClock initialized: now 1781957921000875 us; error 66 us; skew 500 ppm
Jun 20 12:18:41 dist-test-slave-rh5v krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957921, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:41.008013 14629 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:41.009518 14629 webserver.cc:460] Webserver started at http://127.24.141.62:36335/ using document root <none> and password file <none>
I20260620 12:18:41.010267 14629 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:41.010351 14629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:41.010672 14629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:41.013597 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "b69afded77c74064bdd8119e10f7dd8d"
format_stamp: "Formatted at 2026-06-20 12:18:41 on dist-test-slave-rh5v"
server_key: "f0d36d8ed5bf21d6e9ba77bae2808b12"
server_key_iv: "e7cb38b778543a6465478ca9086877e1"
server_key_version: "encryptionkey@0"
I20260620 12:18:41.014340 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "b69afded77c74064bdd8119e10f7dd8d"
format_stamp: "Formatted at 2026-06-20 12:18:41 on dist-test-slave-rh5v"
server_key: "f0d36d8ed5bf21d6e9ba77bae2808b12"
server_key_iv: "e7cb38b778543a6465478ca9086877e1"
server_key_version: "encryptionkey@0"
I20260620 12:18:41.019619 14629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:41.023269 14644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:41.025202 14629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:41.025401 14629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "b69afded77c74064bdd8119e10f7dd8d"
format_stamp: "Formatted at 2026-06-20 12:18:41 on dist-test-slave-rh5v"
server_key: "f0d36d8ed5bf21d6e9ba77bae2808b12"
server_key_iv: "e7cb38b778543a6465478ca9086877e1"
server_key_version: "encryptionkey@0"
I20260620 12:18:41.025553 14629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:41.037279 14629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:41.041643 14629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:41.041888 14629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:41.051173 14629 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:43021
I20260620 12:18:41.051283 14696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:43021 every 8 connection(s)
I20260620 12:18:41.052403 14629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:41.056048 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14629
I20260620 12:18:41.056212 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:41.056557 25140 external_mini_cluster.cc:1428] Setting key daf947a4ff950bfcc3905d90c8aaa138
I20260620 12:18:41.056730 14697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:41.066347 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d: Bootstrap starting.
Jun 20 12:18:41 dist-test-slave-rh5v krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:41.069187 14697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:41.070048 14697 log.cc:824] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:41.072391 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d: No bootstrap required, opened a new log
I20260620 12:18:41.073602 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:41.058260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36721 (local address 127.24.141.62:43021)
0620 12:18:41.058813 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:41.058833 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:41.058876 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:18:41.059676 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:41.059697 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:41.060137 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:41.060492 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:41.061581 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:41.062813 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:41.064159 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:41.064741 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:41.064893 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:41.068189 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:41.068195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:41.068214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:41.068255 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:41.070125 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:41.070636 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:41.070642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:41.070648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:41.070779 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:41.071054 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:41.071057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:41.071059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:41.071607 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:41.071813 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:41.072116 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 12:18:41.072510 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":195,"threads_started":1}
I20260620 12:18:41.076135 14697 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69afded77c74064bdd8119e10f7dd8d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43021 } }
I20260620 12:18:41.076478 14697 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:41.076532 14697 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69afded77c74064bdd8119e10f7dd8d, State: Initialized, Role: FOLLOWER
I20260620 12:18:41.077054 14697 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69afded77c74064bdd8119e10f7dd8d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43021 } }
I20260620 12:18:41.077246 14697 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:41.077317 14697 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:41.077409 14697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:41.077941 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:41.078531 14697 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69afded77c74064bdd8119e10f7dd8d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43021 } }
I20260620 12:18:41.078912 14697 leader_election.cc:302] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69afded77c74064bdd8119e10f7dd8d; no voters:
I20260620 12:18:41.079262 14697 leader_election.cc:288] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:41.079366 14702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:41.079560 14702 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [term 1 LEADER]: Becoming Leader. State: Replica: b69afded77c74064bdd8119e10f7dd8d, State: Running, Role: LEADER
I20260620 12:18:41.079905 14702 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69afded77c74064bdd8119e10f7dd8d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43021 } }
I20260620 12:18:41.080762 14697 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:41.081326 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69afded77c74064bdd8119e10f7dd8d. Latest consensus state: current_term: 1 leader_uuid: "b69afded77c74064bdd8119e10f7dd8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69afded77c74064bdd8119e10f7dd8d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43021 } } }
I20260620 12:18:41.081470 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:41.081759 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69afded77c74064bdd8119e10f7dd8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69afded77c74064bdd8119e10f7dd8d" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 43021 } } }
I20260620 12:18:41.081914 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:41.082938 14710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:41.086520 14710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:41.091711 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:41.082158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57267 (local address 127.24.141.62:43021)
0620 12:18:41.082289 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:41.082293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:41.082640 (+ 347us) server_negotiation.cc:407] Connection header received
0620 12:18:41.082831 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:41.082834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:41.082885 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:41.082938 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:41.084028 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:41.084688 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:41.085926 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:41.086310 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:41.086475 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:41.088310 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:41.088316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:41.088319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:41.088357 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:41.089957 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:41.090470 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:41.090474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:41.090476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:41.090562 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:41.090811 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:41.090815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:41.090816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:41.091034 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:41.091156 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:41.091394 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 12:18:41.091524 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:18:41.095624 14710 catalog_manager.cc:1347] Generated new cluster ID: 96b96b4442964554a2de9ca6e0234f63
I20260620 12:18:41.095821 14710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:41.108623 14710 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:41.109870 14710 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:41.118083 14710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b69afded77c74064bdd8119e10f7dd8d: Generated new TSK 0
I20260620 12:18:41.118956 14710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:41.124662 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14629
2026-06-20T12:18:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:41 dist-test-slave-rh5v krb5kdc[14728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:41 dist-test-slave-rh5v krb5kdc[14728](info): set up 2 sockets
Jun 20 12:18:41 dist-test-slave-rh5v krb5kdc[14728](info): commencing operation
krb5kdc: starting...
W20260620 12:18:42.109880 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:42 dist-test-slave-rh5v krb5kdc[14728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:42.248992 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38607
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:38713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:38607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:42.378273 14741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:42.378603 14741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:42.378650 14741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:42.383054 14741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:42.383147 14741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:42.383167 14741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:42.383184 14741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:42.383199 14741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:42.389113 14741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:38713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:38607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:42.390386 14741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:42.391625 14741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:42.401871 14747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:42.401877 14746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:42.401871 14749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:42.402383 14741 server_base.cc:1034] running on GCE node
I20260620 12:18:42.403175 14741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:42.404767 14741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:42.405972 14741 hybrid_clock.cc:648] HybridClock initialized: now 1781957922405920 us; error 64 us; skew 500 ppm
Jun 20 12:18:42 dist-test-slave-rh5v krb5kdc[14728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957922, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:42.409554 14741 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:42.410794 14741 webserver.cc:460] Webserver started at http://127.24.141.62:44467/ using document root <none> and password file <none>
I20260620 12:18:42.411439 14741 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:42.411509 14741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:42.411700 14741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:42.414083 14741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "d3935f727b41452eb67769b094c1d476"
format_stamp: "Formatted at 2026-06-20 12:18:42 on dist-test-slave-rh5v"
server_key: "8bdecc23424fb2b023a9411b0e0e9707"
server_key_iv: "32193030857e148f4dfcda95725bd812"
server_key_version: "encryptionkey@0"
I20260620 12:18:42.414718 14741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "d3935f727b41452eb67769b094c1d476"
format_stamp: "Formatted at 2026-06-20 12:18:42 on dist-test-slave-rh5v"
server_key: "8bdecc23424fb2b023a9411b0e0e9707"
server_key_iv: "32193030857e148f4dfcda95725bd812"
server_key_version: "encryptionkey@0"
I20260620 12:18:42.419482 14741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 12:18:42.422955 14756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:42.424995 14741 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 12:18:42.425293 14741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "d3935f727b41452eb67769b094c1d476"
format_stamp: "Formatted at 2026-06-20 12:18:42 on dist-test-slave-rh5v"
server_key: "8bdecc23424fb2b023a9411b0e0e9707"
server_key_iv: "32193030857e148f4dfcda95725bd812"
server_key_version: "encryptionkey@0"
I20260620 12:18:42.425454 14741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:42.445007 14741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:42.448786 14741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:42.448975 14741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:42.457199 14741 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:38607
I20260620 12:18:42.457257 14808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:38607 every 8 connection(s)
I20260620 12:18:42.458372 14741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:42.462348 14809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:42.468142 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14741
I20260620 12:18:42.468310 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:42.468772 25140 external_mini_cluster.cc:1428] Setting key a1f4e6096865989a09836b312424bd2d
I20260620 12:18:42.470114 14809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476: Bootstrap starting.
I20260620 12:18:42.473611 14809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:42.474696 14809 log.cc:824] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:42.477536 14809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476: No bootstrap required, opened a new log
I20260620 12:18:42.480331 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:42.470912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35525 (local address 127.24.141.62:38607)
0620 12:18:42.471514 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:42.471541 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:42.471602 (+ 61us) server_negotiation.cc:407] Connection header received
0620 12:18:42.472604 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:42.472645 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:42.473242 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:42.473752 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:42.474295 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:42.475678 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:42.476686 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:42.477284 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:42.477442 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:42.478318 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:42.478325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:42.478341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:42.478843 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:42.478905 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:42.478932 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:42.479024 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:42.479160 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 12:18:42.479418 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":173,"threads_started":1}
I20260620 12:18:42.481767 14809 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3935f727b41452eb67769b094c1d476" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38607 } }
I20260620 12:18:42.482084 14809 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:42.482148 14809 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3935f727b41452eb67769b094c1d476, State: Initialized, Role: FOLLOWER
I20260620 12:18:42.482658 14809 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3935f727b41452eb67769b094c1d476" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38607 } }
I20260620 12:18:42.482859 14809 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:42.482939 14809 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:42.483047 14809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:42.483690 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:42.484375 14809 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3935f727b41452eb67769b094c1d476" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38607 } }
I20260620 12:18:42.484836 14809 leader_election.cc:302] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3935f727b41452eb67769b094c1d476; no voters:
I20260620 12:18:42.485301 14809 leader_election.cc:288] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:42.485749 14819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:42.486016 14819 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [term 1 LEADER]: Becoming Leader. State: Replica: d3935f727b41452eb67769b094c1d476, State: Running, Role: LEADER
I20260620 12:18:42.486515 14819 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3935f727b41452eb67769b094c1d476" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38607 } }
I20260620 12:18:42.486951 14809 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:42.488638 14820 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3935f727b41452eb67769b094c1d476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3935f727b41452eb67769b094c1d476" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38607 } } }
I20260620 12:18:42.488806 14820 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:42.489311 14831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:42.489286 14816 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3935f727b41452eb67769b094c1d476. Latest consensus state: current_term: 1 leader_uuid: "d3935f727b41452eb67769b094c1d476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3935f727b41452eb67769b094c1d476" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 38607 } } }
I20260620 12:18:42.489400 14816 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:42.491899 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:42.486672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48759 (local address 127.24.141.62:38607)
0620 12:18:42.486811 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:42.486817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:42.486838 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:18:42.486893 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:42.486896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:42.486955 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:42.487064 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:42.487410 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:42.488207 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:42.489543 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:42.489872 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:42.489976 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:42.491133 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:42.491138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:42.491140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:42.491158 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:42.491195 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:42.491210 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:42.491282 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:42.491658 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 12:18:42.491725 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:18:42.493902 14831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:42.502611 14831 catalog_manager.cc:1347] Generated new cluster ID: b42405d52d014c3894907ce46c4ec65c
I20260620 12:18:42.502732 14831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:42.521377 14831 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:42.522701 14831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:42.538214 14831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3935f727b41452eb67769b094c1d476: Generated new TSK 0
I20260620 12:18:42.539533 14831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:42.547946 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14741
2026-06-20T12:18:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:42 dist-test-slave-rh5v krb5kdc[14840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:42 dist-test-slave-rh5v krb5kdc[14840](info): set up 2 sockets
Jun 20 12:18:42 dist-test-slave-rh5v krb5kdc[14840](info): commencing operation
krb5kdc: starting...
W20260620 12:18:43.521874 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:43 dist-test-slave-rh5v krb5kdc[14840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:43.656329 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:41225
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:43.786458 14853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:43.786808 14853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:43.786854 14853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:43.791198 14853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:43.791292 14853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:43.791311 14853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:43.791329 14853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:43.791348 14853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:43.797606 14853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:41225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:43.799062 14853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:43.800341 14853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:43.810655 14858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:43.810698 14853 server_base.cc:1034] running on GCE node
W20260620 12:18:43.810655 14859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:43.810683 14861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:43.811697 14853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:43.813216 14853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:43.814393 14853 hybrid_clock.cc:648] HybridClock initialized: now 1781957923814378 us; error 43 us; skew 500 ppm
Jun 20 12:18:43 dist-test-slave-rh5v krb5kdc[14840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:43.817963 14853 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:43.819514 14853 webserver.cc:460] Webserver started at http://127.24.141.62:45301/ using document root <none> and password file <none>
I20260620 12:18:43.820425 14853 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:43.820550 14853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:43.820928 14853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:43.823575 14853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "ac2c3a53dece4ce183d22689ef36f672"
format_stamp: "Formatted at 2026-06-20 12:18:43 on dist-test-slave-rh5v"
server_key: "dc28ceecde25911ae0deec939807416e"
server_key_iv: "ad09cd6eb0ccf1d5094583e7ff4ba074"
server_key_version: "encryptionkey@0"
I20260620 12:18:43.824314 14853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "ac2c3a53dece4ce183d22689ef36f672"
format_stamp: "Formatted at 2026-06-20 12:18:43 on dist-test-slave-rh5v"
server_key: "dc28ceecde25911ae0deec939807416e"
server_key_iv: "ad09cd6eb0ccf1d5094583e7ff4ba074"
server_key_version: "encryptionkey@0"
I20260620 12:18:43.829166 14853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 12:18:43.832336 14868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:43.834156 14853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 12:18:43.834362 14853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "ac2c3a53dece4ce183d22689ef36f672"
format_stamp: "Formatted at 2026-06-20 12:18:43 on dist-test-slave-rh5v"
server_key: "dc28ceecde25911ae0deec939807416e"
server_key_iv: "ad09cd6eb0ccf1d5094583e7ff4ba074"
server_key_version: "encryptionkey@0"
I20260620 12:18:43.834508 14853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:43.839771 14853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:43.843336 14853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:43.843544 14853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:43.851673 14853 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:41225
I20260620 12:18:43.851769 14920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:41225 every 8 connection(s)
I20260620 12:18:43.852835 14853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:43.854913 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14853
I20260620 12:18:43.855098 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:43.855455 25140 external_mini_cluster.cc:1428] Setting key f602e4c6f40fbb30caf4c6b9b22d6b44
I20260620 12:18:43.857441 14921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:43.862776 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:43.857596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57565 (local address 127.24.141.62:41225)
0620 12:18:43.858339 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:43.858359 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:43.858416 (+ 57us) server_negotiation.cc:407] Connection header received
0620 12:18:43.859190 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:43.859224 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:43.859624 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:43.860054 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:43.861254 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:43.861259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:43.861277 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:43.861291 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:43.861361 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:43.861385 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:43.861550 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:43.861633 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 12:18:43.861848 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":225,"threads_started":1}
I20260620 12:18:43.866088 14921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672: Bootstrap starting.
I20260620 12:18:43.866889 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:43.869386 14921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:43.870380 14921 log.cc:824] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:43.871555 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:43.869499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42165 (local address 127.24.141.62:41225)
0620 12:18:43.869654 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:43.869659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:43.869905 (+ 246us) server_negotiation.cc:407] Connection header received
0620 12:18:43.870075 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:43.870078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:43.870122 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:43.870187 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:43.871085 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:43.871089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:43.871092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:18:43.871106 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:43.871141 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:18:43.871154 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:43.871240 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:43.871377 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 12:18:43.871420 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:18:43.873510 14921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672: No bootstrap required, opened a new log
I20260620 12:18:43.877763 14921 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2c3a53dece4ce183d22689ef36f672" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41225 } }
I20260620 12:18:43.878089 14921 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:18:43.878145 14921 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2c3a53dece4ce183d22689ef36f672, State: Initialized, Role: FOLLOWER
I20260620 12:18:43.878618 14921 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2c3a53dece4ce183d22689ef36f672" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41225 } }
I20260620 12:18:43.878769 14921 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:18:43.878839 14921 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:18:43.878953 14921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:43.880087 14921 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2c3a53dece4ce183d22689ef36f672" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41225 } }
I20260620 12:18:43.880491 14921 leader_election.cc:302] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac2c3a53dece4ce183d22689ef36f672; no voters:
I20260620 12:18:43.880877 14921 leader_election.cc:288] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:18:43.881131 14931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:43.881551 14931 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [term 1 LEADER]: Becoming Leader. State: Replica: ac2c3a53dece4ce183d22689ef36f672, State: Running, Role: LEADER
I20260620 12:18:43.882093 14931 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2c3a53dece4ce183d22689ef36f672" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41225 } }
I20260620 12:18:43.882761 14921 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:43.884244 14932 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac2c3a53dece4ce183d22689ef36f672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2c3a53dece4ce183d22689ef36f672" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41225 } } }
I20260620 12:18:43.884476 14932 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:43.884774 14933 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac2c3a53dece4ce183d22689ef36f672. Latest consensus state: current_term: 1 leader_uuid: "ac2c3a53dece4ce183d22689ef36f672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2c3a53dece4ce183d22689ef36f672" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41225 } } }
I20260620 12:18:43.885000 14933 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:43.885177 14941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:43.889843 14941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:43.897522 14941 catalog_manager.cc:1347] Generated new cluster ID: d5586a040a344c8e8035afa4a738feec
I20260620 12:18:43.897642 14941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:43.920217 14941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:43.920692 14941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:43.935559 14941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac2c3a53dece4ce183d22689ef36f672: Generated new TSK 0
I20260620 12:18:43.936559 14941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:18:43.942793 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14853
2026-06-20T12:18:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1392 ms)
[----------] 18 tests from AuthTokenIssuingTest (25485 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:44 dist-test-slave-rh5v krb5kdc[14952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:44 dist-test-slave-rh5v krb5kdc[14952](info): set up 2 sockets
Jun 20 12:18:44 dist-test-slave-rh5v krb5kdc[14952](info): commencing operation
krb5kdc: starting...
W20260620 12:18:44.909951 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:44 dist-test-slave-rh5v krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:45.049351 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:41025
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:41025,127.24.141.61:35575,127.24.141.60:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:45.182871 14965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:18:45.183290 14965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:45.183317 14965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:45.183362 14965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:45.187884 14965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:45.187990 14965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:45.188009 14965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:45.188026 14965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:45.188041 14965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:45.194399 14965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:41025,127.24.141.61:35575,127.24.141.60:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:41025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:45.195969 14965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:45.197340 14965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:45.208539 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:45.208525 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:45.208525 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:45.208613 14965 server_base.cc:1034] running on GCE node
I20260620 12:18:45.209597 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:45.211118 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:45.212359 14965 hybrid_clock.cc:648] HybridClock initialized: now 1781957925212328 us; error 70 us; skew 500 ppm
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:45.216356 14965 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:45.217849 14965 webserver.cc:460] Webserver started at http://127.24.141.62:43327/ using document root <none> and password file <none>
I20260620 12:18:45.218600 14965 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:45.218679 14965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:45.218928 14965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:45.221614 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "cdcfb036791b467d8a5d201ac7637daf"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "0a4fcd7e4f97aaa95d1ce4fa0b3ecda4"
server_key_iv: "7cb43bece8d174c10ccd4c09edf85332"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.222276 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "cdcfb036791b467d8a5d201ac7637daf"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "0a4fcd7e4f97aaa95d1ce4fa0b3ecda4"
server_key_iv: "7cb43bece8d174c10ccd4c09edf85332"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.227483 14965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 12:18:45.231179 14980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:45.233168 14965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:18:45.233408 14965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "cdcfb036791b467d8a5d201ac7637daf"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "0a4fcd7e4f97aaa95d1ce4fa0b3ecda4"
server_key_iv: "7cb43bece8d174c10ccd4c09edf85332"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.233592 14965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:45.256978 14965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:45.261759 14965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:45.262042 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:45.271513 14965 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:41025
I20260620 12:18:45.271842 15032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:41025 every 8 connection(s)
I20260620 12:18:45.272903 14965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:45.276923 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:45.278932 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 14965
I20260620 12:18:45.279104 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:45.279439 25140 external_mini_cluster.cc:1428] Setting key 2065e75465bd80837736ced02114e78e
I20260620 12:18:45.282436 15033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.141.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.61@KRBTEST.COM" created.
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:45.304796 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.285827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:41025 (local address 127.0.0.1:48394)
0620 12:18:45.286537 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.286712 (+ 175us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.288111 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.289564 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.289572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.290135 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.290623 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.290645 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.292240 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.292245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.293591 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.293615 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.294537 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.294542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.294664 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.295381 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.295411 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.297448 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.300387 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.300402 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.300421 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.300881 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.301461 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.301465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.301467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.301662 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.302736 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.302744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.302918 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.303148 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.303584 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":186,"threads_started":1}
I20260620 12:18:45.304793 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.285829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.24.141.62:41025)
0620 12:18:45.286651 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.286679 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.286900 (+ 221us) server_negotiation.cc:407] Connection header received
0620 12:18:45.288731 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.288740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.289199 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.289489 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.290847 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.291979 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.293804 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.294315 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.294528 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.297747 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.297757 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.297774 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.297820 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.300193 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.301133 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.301143 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.301144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.301340 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.301840 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.301844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.301846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.302599 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.302749 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.303402 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.303747 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":633,"thread_start_us":197,"threads_started":1}
I20260620 12:18:45.308226 15033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:45.309480 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.308978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45838)
0620 12:18:45.309272 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.309372 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
W20260620 12:18:45.309732 14984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
I20260620 12:18:45.311091 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.310793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45842)
0620 12:18:45.310953 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.311015 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 12:18:45.311259 15033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:35575: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab.
Entry for principal kudu/127.24.141.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab.
I20260620 12:18:45.312979 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:35575
--webserver_interface=127.24.141.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab
--principal=kudu/127.24.141.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:41025,127.24.141.61:35575,127.24.141.60:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:18:45.360756 15033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } attempt: 1
I20260620 12:18:45.361747 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.361430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45850)
0620 12:18:45.361589 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.361653 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 12:18:45.363021 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.362672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45858)
0620 12:18:45.362871 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.362946 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 12:18:45.363212 15033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:35575: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
I20260620 12:18:45.431394 15033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } attempt: 2
I20260620 12:18:45.432307 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.431991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45868)
0620 12:18:45.432159 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.432220 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 12:18:45.433405 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.433104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45876)
0620 12:18:45.433269 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.433334 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 12:18:45.433579 15033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:35575: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
W20260620 12:18:45.453553 15040 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:18:45.454020 15040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:45.454051 15040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:45.454129 15040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:45.459024 15040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:45.459160 15040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:45.459192 15040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:45.459215 15040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:45.459254 15040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:45.468585 15040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:41025,127.24.141.61:35575,127.24.141.60:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab
--principal=kudu/127.24.141.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.141.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:45.470558 15040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:45.472081 15040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:45.483479 15046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:45.483618 15040 server_base.cc:1034] running on GCE node
W20260620 12:18:45.483538 15045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:45.483479 15048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:45.484521 15040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:45.485991 15040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:45.487247 15040 hybrid_clock.cc:648] HybridClock initialized: now 1781957925487217 us; error 55 us; skew 500 ppm
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:45.490865 15040 init.cc:377] Logged in from keytab as kudu/127.24.141.61@KRBTEST.COM (short username kudu)
I20260620 12:18:45.492125 15040 webserver.cc:460] Webserver started at http://127.24.141.61:44713/ using document root <none> and password file <none>
I20260620 12:18:45.492815 15040 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:45.492890 15040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:45.493165 15040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:45.495627 15040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data/instance:
uuid: "3e2b8b2027ff4f0ea3c720951e200992"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "74796c44f880bdb07e6c22d6ccd2bdb9"
server_key_iv: "3f1d524a6d600fa37d954846199476d2"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.496304 15040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/wal/instance:
uuid: "3e2b8b2027ff4f0ea3c720951e200992"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "74796c44f880bdb07e6c22d6ccd2bdb9"
server_key_iv: "3f1d524a6d600fa37d954846199476d2"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.501368 15040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:18:45.505117 15055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:45.507215 15040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:45.507432 15040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/wal
uuid: "3e2b8b2027ff4f0ea3c720951e200992"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "74796c44f880bdb07e6c22d6ccd2bdb9"
server_key_iv: "3f1d524a6d600fa37d954846199476d2"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.507599 15040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:45.524782 15033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } attempt: 3
I20260620 12:18:45.525965 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.525607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45892)
0620 12:18:45.525794 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.525868 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 12:18:45.526965 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.526716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45902)
0620 12:18:45.526850 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.526904 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 12:18:45.527137 15033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:35575: Network error: Client connection negotiation failed: client connection to 127.24.141.61:35575: connect: Connection refused (error 111)
I20260620 12:18:45.531509 15040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:45.535322 15040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:45.535516 15040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:45.544524 15040 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.61:35575
I20260620 12:18:45.544873 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.61:35575 every 8 connection(s)
I20260620 12:18:45.545779 15040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
I20260620 12:18:45.549823 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:45.552956 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 15040
I20260620 12:18:45.553130 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-1/wal/instance
I20260620 12:18:45.553498 25140 external_mini_cluster.cc:1428] Setting key 5e53466ed2aa979a544608fce6f89793
I20260620 12:18:45.555541 15108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.141.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.60@KRBTEST.COM" created.
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:45.575173 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.558539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.24.141.62:41025)
0620 12:18:45.558702 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.558707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.560250 (+ 1543us) server_negotiation.cc:407] Connection header received
0620 12:18:45.561573 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.561577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.561649 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.561733 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.563208 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.563806 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.565964 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.566257 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.566382 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.570242 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.570246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.570250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.570281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.571740 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.572760 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.572763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.572764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.572847 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.573431 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.573434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.573435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.573715 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.573821 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.574759 (+ 938us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.574946 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:18:45.576839 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.559156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:41025 (local address 127.0.0.1:48408)
0620 12:18:45.560011 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.560075 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.561300 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.561959 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.561967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.562392 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.563042 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.563059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.564139 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.564147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.565752 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.565779 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.566488 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.566493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.566726 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.567456 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.567480 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.569959 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.571974 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.571985 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.572011 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.572587 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.572967 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.572971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.572973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.573257 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.573876 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.573884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.574364 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.574634 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.575075 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":226,"threads_started":1}
I20260620 12:18:45.578701 15108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab.
Entry for principal kudu/127.24.141.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab.
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:18:45.586561 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:40745
--webserver_interface=127.24.141.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab
--principal=kudu/127.24.141.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:41025,127.24.141.61:35575,127.24.141.60:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:18:45.590256 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.579321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45910)
0620 12:18:45.579565 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.579583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.579711 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.580467 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.580471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.580639 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.580767 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.580775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.582079 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.582083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.582758 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.582775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.583622 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.583626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.583737 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.584387 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.584399 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.585901 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.588221 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.588225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.588228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.588496 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.588918 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.588921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.588923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.589030 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.589715 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.589718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.589836 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.589932 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.590094 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 12:18:45.590484 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.579554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.24.141.61:35575)
0620 12:18:45.579994 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.580006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.580027 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:18:45.580073 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.580079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.580289 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.580425 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.580876 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.581875 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.582952 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.583430 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.583525 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.586151 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.586168 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.586187 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.586231 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.588047 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.588632 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.588641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.588643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.588780 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.589171 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.589174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.589176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.589584 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.589743 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.590062 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.590269 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":165,"threads_started":1}
I20260620 12:18:45.593254 15108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:45.594301 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.593941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57524)
0620 12:18:45.594117 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.594204 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 12:18:45.594543 15059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
I20260620 12:18:45.595574 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.595335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57536)
0620 12:18:45.595448 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.595504 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 12:18:45.595752 15108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:40745: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
I20260620 12:18:45.645164 15108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } attempt: 1
I20260620 12:18:45.646374 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.645940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57550)
0620 12:18:45.646165 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.646259 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 12:18:45.647306 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.647082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57554)
0620 12:18:45.647187 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.647240 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:18:45.647470 15108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:40745: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
I20260620 12:18:45.670310 15033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } attempt: 4
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:18:45.680238 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.671259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45924)
0620 12:18:45.671535 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.671604 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.671742 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.672007 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.672010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.672248 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.672390 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.672397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.673290 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.673294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.674367 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.674377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.674871 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.674875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.674983 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.675530 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.675551 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.676887 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.678705 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.678709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.678711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.678906 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.679167 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.679171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.679173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.679256 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.679644 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.679647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.679756 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.679849 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.680019 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 12:18:45.680238 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.671360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.24.141.61:35575)
0620 12:18:45.671546 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.671551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.671677 (+ 126us) server_negotiation.cc:407] Connection header received
0620 12:18:45.671848 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.671851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.671901 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.671974 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.672519 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.673175 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.674493 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.674756 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.674849 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.677034 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.677037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.677040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.677110 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.678593 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.678998 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.679002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.679004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.679085 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.679334 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.679337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.679339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.679546 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.679645 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.679909 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.680038 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 12:18:45.681289 15033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:45.682250 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.681976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57564)
0620 12:18:45.682105 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.682173 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 12:18:45.683112 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.682920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57574)
0620 12:18:45.683016 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.683063 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 12:18:45.683250 15033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:40745: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
I20260620 12:18:45.715626 15108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } attempt: 2
I20260620 12:18:45.716766 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.716274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57576)
0620 12:18:45.716514 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.716627 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 12:18:45.717727 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.717454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57588)
0620 12:18:45.717568 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.717639 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 12:18:45.717900 15108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:40745: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
W20260620 12:18:45.732188 15114 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:18:45.732533 15114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:45.732558 15114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:45.732609 15114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:45.736960 15114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:45.737094 15114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:45.737129 15114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:45.737151 15114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:45.737166 15114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:45.742424 15033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } attempt: 1
I20260620 12:18:45.743625 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.743171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57596)
0620 12:18:45.743373 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.743463 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 12:18:45.744061 15114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:41025,127.24.141.61:35575,127.24.141.60:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab
--principal=kudu/127.24.141.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.141.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:45.744495 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.744223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57604)
0620 12:18:45.744361 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.744430 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 12:18:45.744776 15033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:40745: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
I20260620 12:18:45.745558 15114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:45.746731 15114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:45.757102 15120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:45.757109 15121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:45.757102 15123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:45.757628 15114 server_base.cc:1034] running on GCE node
I20260620 12:18:45.758423 15114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:45.760071 15114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:45.763439 15114 hybrid_clock.cc:648] HybridClock initialized: now 1781957925763366 us; error 74 us; skew 500 ppm
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:45.767479 15114 init.cc:377] Logged in from keytab as kudu/127.24.141.60@KRBTEST.COM (short username kudu)
I20260620 12:18:45.768800 15114 webserver.cc:460] Webserver started at http://127.24.141.60:42313/ using document root <none> and password file <none>
I20260620 12:18:45.769582 15114 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:45.769661 15114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:45.769879 15114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:45.772506 15114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data/instance:
uuid: "9f3af093d9214f3a9d94e5bd15b2bf41"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "cc3fa4a910e95011187996004ddf162c"
server_key_iv: "d302e693cfc1331d3e3953478ef05cfc"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.773211 15114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/wal/instance:
uuid: "9f3af093d9214f3a9d94e5bd15b2bf41"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "cc3fa4a910e95011187996004ddf162c"
server_key_iv: "d302e693cfc1331d3e3953478ef05cfc"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.778194 15114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 12:18:45.782177 15130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:45.784255 15114 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:18:45.784484 15114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/wal
uuid: "9f3af093d9214f3a9d94e5bd15b2bf41"
format_stamp: "Formatted at 2026-06-20 12:18:45 on dist-test-slave-rh5v"
server_key: "cc3fa4a910e95011187996004ddf162c"
server_key_iv: "d302e693cfc1331d3e3953478ef05cfc"
server_key_version: "encryptionkey@0"
I20260620 12:18:45.784657 15114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:45.805933 15114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:45.809128 15108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } attempt: 3
I20260620 12:18:45.810425 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.809786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57612)
0620 12:18:45.809991 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.810297 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 12:18:45.810966 15114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:45.811245 15114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:45.811607 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.811224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57626)
0620 12:18:45.811441 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.811523 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 12:18:45.811791 15108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:40745: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
I20260620 12:18:45.811933 15033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } attempt: 2
I20260620 12:18:45.813001 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.812590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57638)
0620 12:18:45.812830 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.812893 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260620 12:18:45.814185 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.813825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57644)
0620 12:18:45.813992 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.814088 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 12:18:45.814399 15033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:40745: Network error: Client connection negotiation failed: client connection to 127.24.141.60:40745: connect: Connection refused (error 111)
I20260620 12:18:45.820546 15114 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.60:40745
I20260620 12:18:45.820894 15182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.60:40745 every 8 connection(s)
I20260620 12:18:45.821753 15114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
I20260620 12:18:45.825583 15183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:45.826304 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 15114
I20260620 12:18:45.826449 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957618728254-25140-0/minicluster-data/master-2/wal/instance
I20260620 12:18:45.826807 25140 external_mini_cluster.cc:1428] Setting key e6158e833ac37a3b3253bc2a67f53c06
I20260620 12:18:45.832681 15183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:45.845463 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.828708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.24.141.60:40745)
0620 12:18:45.829467 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.829490 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.829544 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:18:45.830527 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.830551 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.831010 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.831407 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.832381 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.833636 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.834572 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.835193 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.835522 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.838608 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.838618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.838633 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.838688 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.841208 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.841792 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.841800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.841807 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.841966 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.842295 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.842300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.842301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.843072 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.843429 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.843824 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.844221 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":174,"threads_started":1}
I20260620 12:18:45.847113 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.835083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:41025 (local address 127.0.0.1:48416)
0620 12:18:45.836153 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.836201 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.836373 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.836681 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.836687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.837164 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.837330 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.837337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.838278 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.838284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.839434 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.839454 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.840049 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.840054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.840224 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.840777 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.840793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.842551 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.844971 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.844981 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.844987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.845305 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.845686 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.845690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.845693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.845806 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.846292 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.846298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.846456 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.846610 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.846835 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":164,"threads_started":1}
I20260620 12:18:45.847193 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.835180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.24.141.62:41025)
0620 12:18:45.835375 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.835381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.836313 (+ 932us) server_negotiation.cc:407] Connection header received
0620 12:18:45.836476 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.836480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.836532 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.836603 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.837455 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.838042 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.839615 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.839892 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.840001 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.842836 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.842840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.842843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.842878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.844778 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.845416 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.845420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.845422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.845546 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.845933 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.845937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.845938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.846187 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.846275 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.846764 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.847028 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:18:45.848737 15183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:45.860178 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.849675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.24.141.61:35575)
0620 12:18:45.849899 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.849904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.849918 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:18:45.849960 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.849962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.850006 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.850086 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.850697 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.851789 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.853156 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.853560 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.853701 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.856469 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.856475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.856478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.856526 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.858298 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.858863 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.858868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.858869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.858980 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.859254 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.859258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.859260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.859503 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.859595 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.859877 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.860010 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 12:18:45.860178 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.849514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:35575 (local address 127.0.0.1:45934)
0620 12:18:45.849697 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.849713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.849804 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.850137 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.850141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.850338 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.850501 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.850509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.851981 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.851984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.852919 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.852932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.853742 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.853746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.853876 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.854617 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.854630 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.856222 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.858497 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.858501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.858503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.858724 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.859078 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.859082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.859084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.859167 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.859595 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.859598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.859697 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.859790 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.859970 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 12:18:45.861138 15183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:45.861240 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.848971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.24.141.62:41025)
0620 12:18:45.849144 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.849148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.849298 (+ 150us) server_negotiation.cc:407] Connection header received
0620 12:18:45.849433 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.849436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.849502 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.849597 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.850702 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.851789 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.853156 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.853562 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.853714 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.857326 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.857330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.857333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.857370 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.859123 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.859640 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.859643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.859645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.859734 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.859985 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.859988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.859990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.860212 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.860461 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.860876 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.861036 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:18:45.871656 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.861757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57670)
0620 12:18:45.861903 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.861919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.862016 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.862370 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.862373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.862549 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.862670 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.862676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.863705 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.863708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.864482 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.864492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.865205 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.865208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.865279 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.865821 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.865833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.867496 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.869828 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.869831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.869834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.870080 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.870442 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.870445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.870447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.870549 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.871039 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.871042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.871177 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.871304 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.871475 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 12:18:45.871833 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.861848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.24.141.60:40745)
0620 12:18:45.862156 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.862160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.862171 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:18:45.862200 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.862203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.862255 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.862334 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.862785 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.863566 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.864633 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.865011 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.865196 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.867711 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.867716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.867720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.867761 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.869694 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.870215 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.870219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.870221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.870337 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.870652 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.870655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.870657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.870896 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.870981 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.871417 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.871580 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260620 12:18:45.873699 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.862773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.24.141.61:35575)
0620 12:18:45.862909 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.862914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.863129 (+ 215us) server_negotiation.cc:407] Connection header received
0620 12:18:45.863243 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.863245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.863291 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.863364 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.864110 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.864806 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.865892 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.866209 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.866322 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.869627 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.869631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.869634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.869665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.871284 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.871893 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.871898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.871900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.872017 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.872446 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.872449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.872451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.872743 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.873093 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.873337 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.873519 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:18:45.877032 15183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: Bootstrap starting.
I20260620 12:18:45.877760 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:45.880501 15183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:45.879325 15007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cdcfb036791b467d8a5d201ac7637daf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48428
I20260620 12:18:45.881633 15183 log.cc:824] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:45.884595 15183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: No bootstrap required, opened a new log
I20260620 12:18:45.889372 15183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:45.889740 15183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3af093d9214f3a9d94e5bd15b2bf41, State: Initialized, Role: FOLLOWER
I20260620 12:18:45.890157 15183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:45.890936 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:45.891181 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:45.891754 15183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 12:18:45.897723 15200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:45.897873 15200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:45.896545 15114 master_runner.cc:186] Error getting master registration for 127.24.141.62:41025: OK, instance_id { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" instance_seqno: 1781957925268779 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:18:45.901544 15114 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:18:45.914606 15033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } attempt: 3
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
I20260620 12:18:45.925665 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.915511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.24.141.60:40745)
0620 12:18:45.915660 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.915665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.915687 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:18:45.915842 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.915846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.915913 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.916041 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.916518 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.917450 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.918660 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.919024 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.919146 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.922006 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.922012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.922016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.922057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.923593 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.924228 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.924232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.924234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.924318 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.924687 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.924690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.924692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.924917 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.925014 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.925320 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.925455 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:18:45.925665 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.915411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57676)
0620 12:18:45.915590 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.915609 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.915731 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.916058 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.916060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.916276 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.916403 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.916408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.917583 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.917586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.918520 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.918530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.919162 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.919165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.919263 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.919871 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.919887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.921703 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.923736 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.923740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.923743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.924094 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.924483 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.924487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.924489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.924591 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.925019 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.925022 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.925166 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.925259 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.925461 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 12:18:45.930231 15033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf: Bootstrap starting.
I20260620 12:18:45.933661 15033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:45.934712 15033 log.cc:824] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:45.937848 15033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf: No bootstrap required, opened a new log
I20260620 12:18:45.941540 15033 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:45.941867 15033 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcfb036791b467d8a5d201ac7637daf, State: Initialized, Role: FOLLOWER
I20260620 12:18:45.942260 15033 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:45.943059 15203 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:45.943315 15203 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:45.944023 15033 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:45.944111 15007 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 12:18:45.944283 15007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:45.945961 15007 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:45.949275 15157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdcfb036791b467d8a5d201ac7637daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41"
I20260620 12:18:45.949661 15157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:45.951713 15157 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcfb036791b467d8a5d201ac7637daf in term 1.
I20260620 12:18:45.952487 14984 leader_election.cc:302] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f3af093d9214f3a9d94e5bd15b2bf41, cdcfb036791b467d8a5d201ac7637daf; no voters:
I20260620 12:18:45.952934 15203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [term 1 FOLLOWER]: Leader election won for term 1
W20260620 12:18:45.953249 15214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 12:18:45.953254 15203 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [term 1 LEADER]: Becoming Leader. State: Replica: cdcfb036791b467d8a5d201ac7637daf, State: Running, Role: LEADER
W20260620 12:18:45.953438 15214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:45.953872 15203 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:45.950649 14965 master_runner.cc:186] Error getting master registration for 127.24.141.61:35575: OK, instance_id { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" instance_seqno: 1781957925542157 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:18:45.954365 14965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:18:45.954967 15108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } attempt: 4
I20260620 12:18:45.956295 15203 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdcfb036791b467d8a5d201ac7637daf. Latest consensus state: current_term: 1 leader_uuid: "cdcfb036791b467d8a5d201ac7637daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:45.956482 15203 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:45.953375 15007 leader_election.cc:288] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [CANDIDATE]: Term 1 election: Requested vote from peers 3e2b8b2027ff4f0ea3c720951e200992 (127.24.141.61:35575), 9f3af093d9214f3a9d94e5bd15b2bf41 (127.24.141.60:40745)
I20260620 12:18:45.961884 15218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 12:18:45 dist-test-slave-rh5v krb5kdc[14952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957925, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
I20260620 12:18:45.963019 15218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:45.968319 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.956122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.24.141.60:40745)
0620 12:18:45.956299 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.956303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.956318 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:18:45.956365 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.956368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.956416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.957042 (+ 626us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.957186 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.957968 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.959137 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.959472 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.959570 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.963095 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.963101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.963105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.963148 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.965350 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.966315 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.966321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.966323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.966448 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.966960 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.966964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.966965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.967280 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.967373 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.967933 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.968107 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 12:18:45.969411 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.955502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:40745 (local address 127.0.0.1:57684)
0620 12:18:45.955851 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:45.955867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:45.955964 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:45.956548 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:45.956552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:45.956780 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:45.956926 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.956934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.958125 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.958129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.958978 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:45.958989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.959691 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.959697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:45.959838 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.960634 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:45.960651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:45.962694 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:45.965627 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.965631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.965634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.966129 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.966631 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:45.966634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:45.966636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:45.966803 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:45.967471 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:45.967474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:45.967639 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 12:18:45.968764 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0620 12:18:45.969111 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
I20260620 12:18:45.971032 15156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [term 1 FOLLOWER]: Refusing update from remote peer cdcfb036791b467d8a5d201ac7637daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:45.972085 15216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:45.976488 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: Bootstrap starting.
I20260620 12:18:45.983700 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdcfb036791b467d8a5d201ac7637daf. Latest consensus state: current_term: 1 leader_uuid: "cdcfb036791b467d8a5d201ac7637daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:45.983924 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:45.986008 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.965576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.24.141.62:41025)
0620 12:18:45.965700 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.965705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.971231 (+ 5526us) server_negotiation.cc:407] Connection header received
0620 12:18:45.971411 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.971415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.971469 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.971522 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.975427 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.976202 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.978668 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.979110 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.979217 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.981448 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.981454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.981457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.981499 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.983560 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.984208 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.984224 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.984226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.984334 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.984665 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.984669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.984671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.984929 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.985050 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.985455 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.985602 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:18:45.991248 15108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:45.999812 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.966905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.24.141.60:40745)
0620 12:18:45.968732 (+ 1827us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.968737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.984250 (+ 15513us) server_negotiation.cc:407] Connection header received
0620 12:18:45.984500 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.984504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.984762 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.985218 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.986332 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.987761 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.989347 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.989854 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.990004 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.992507 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.992515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.992519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.992610 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.995683 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.996469 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.996473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.996476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.996648 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:45.997696 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:45.997700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:45.997702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:45.998191 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:45.999136 (+ 945us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:45.999272 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 12:18:45.999498 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1715,"thread_start_us":105,"threads_started":1}
I20260620 12:18:46.001924 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdcfb036791b467d8a5d201ac7637daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:46.002167 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:46.002823 15228 mvcc.cc:205] Tried to move back new op lower bound from 7298899664763625472 to 7298899664711634944. Current Snapshot: MvccSnapshot[applied={T|T < 7298899664763625472}]
I20260620 12:18:46.002918 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdcfb036791b467d8a5d201ac7637daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:46.003090 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:46.005016 15108 log.cc:824] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:46.008883 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:45.966127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.24.141.61:35575)
0620 12:18:45.966244 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:45.966248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:45.969572 (+ 3324us) server_negotiation.cc:407] Connection header received
0620 12:18:45.969776 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:45.969779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:45.969849 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:45.969904 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:45.972863 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.973940 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.977522 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:45.977888 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:45.977984 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:45.996710 (+ 18726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:45.996716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:45.996719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:45.996765 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:45.998827 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:46.005755 (+ 6928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:46.005762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:46.005764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:46.005940 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:46.006977 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:46.006982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:46.006984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:46.007301 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:46.007477 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:46.008488 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0620 12:18:46.008673 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:18:46.011451 15218 catalog_manager.cc:1347] Generated new cluster ID: d9e5bd50a90e4f97819f6ea06d92c46d
I20260620 12:18:46.011555 15218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:18:46.013216 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: No bootstrap required, opened a new log
I20260620 12:18:46.022234 15108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:46.022883 15108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2b8b2027ff4f0ea3c720951e200992, State: Initialized, Role: FOLLOWER
I20260620 12:18:46.023633 15108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } }
I20260620 12:18:46.024763 15233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:46.025261 15233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:46.026893 15081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdcfb036791b467d8a5d201ac7637daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3e2b8b2027ff4f0ea3c720951e200992"
I20260620 12:18:46.027235 15081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:46.029620 15081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcfb036791b467d8a5d201ac7637daf in term 1.
I20260620 12:18:46.030565 15082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [term 1 FOLLOWER]: Refusing update from remote peer cdcfb036791b467d8a5d201ac7637daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:46.031641 15108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:46.032433 15203 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:18:46.037755 15061 tablet.cc:2369] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:18:46.048342 15244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:46.048487 15244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:46.053617 15218 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:46.055266 15233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdcfb036791b467d8a5d201ac7637daf. Latest consensus state: current_term: 1 leader_uuid: "cdcfb036791b467d8a5d201ac7637daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:46.055464 15233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:46.055550 15233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdcfb036791b467d8a5d201ac7637daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:46.055613 15233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:46.055709 15245 mvcc.cc:205] Tried to move back new op lower bound from 7298899664763625472 to 7298899664711634944. Current Snapshot: MvccSnapshot[applied={T|T < 7298899664763625472}]
I20260620 12:18:46.055898 15218 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:46.061007 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdcfb036791b467d8a5d201ac7637daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcfb036791b467d8a5d201ac7637daf" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 41025 } } peers { permanent_uuid: "3e2b8b2027ff4f0ea3c720951e200992" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 35575 } } peers { permanent_uuid: "9f3af093d9214f3a9d94e5bd15b2bf41" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 40745 } } }
I20260620 12:18:46.061237 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:46.089160 15218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdcfb036791b467d8a5d201ac7637daf: Generated new TSK 0
I20260620 12:18:46.090278 15218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:18:46.139038 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.135791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.24.141.61:35575)
0620 12:18:46.135952 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.135957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.137408 (+ 1451us) server_negotiation.cc:407] Connection header received
0620 12:18:46.137904 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.137908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.137973 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.138035 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.138874 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45956: BlockingRecv error: recv got EOF from 127.0.0.1:45956 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 12:18:46.140306 15186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.136634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.24.141.60:40745)
0620 12:18:46.136753 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.136758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.137769 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 12:18:46.139477 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.139481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.139534 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.139583 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.140150 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57694: BlockingRecv error: recv got EOF from 127.0.0.1:57694 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 12:18:46.142614 15037 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.141215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.24.141.62:41025)
0620 12:18:46.141335 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.141339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.141650 (+ 311us) server_negotiation.cc:407] Connection header received
0620 12:18:46.141815 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.141818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.141866 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.141912 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.142463 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48448: BlockingRecv error: recv got EOF from 127.0.0.1:48448 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 12:18:46.147598 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.145726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.24.141.61:35575)
0620 12:18:46.145881 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.145886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.146491 (+ 605us) server_negotiation.cc:407] Connection header received
0620 12:18:46.146736 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.146740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.146824 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.146878 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.147467 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45966: BlockingRecv error: recv got EOF from 127.0.0.1:45966 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 12:18:46.153409 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.151458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.24.141.61:35575)
0620 12:18:46.151588 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.151593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.152199 (+ 606us) server_negotiation.cc:407] Connection header received
0620 12:18:46.152469 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.152473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.152527 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.152577 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.153269 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45976: BlockingRecv error: recv got EOF from 127.0.0.1:45976 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 12:18:46.160393 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.158362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.24.141.61:35575)
0620 12:18:46.158508 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.158514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.159250 (+ 736us) server_negotiation.cc:407] Connection header received
0620 12:18:46.159494 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.159498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.159557 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.159610 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.160262 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45980: BlockingRecv error: recv got EOF from 127.0.0.1:45980 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:18:46.168439 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.167047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.24.141.61:35575)
0620 12:18:46.167177 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.167182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.167439 (+ 257us) server_negotiation.cc:407] Connection header received
0620 12:18:46.167572 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.167576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.167628 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.167695 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.168169 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45982: BlockingRecv error: recv got EOF from 127.0.0.1:45982 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 12:18:46.180591 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.179109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.24.141.61:35575)
0620 12:18:46.179355 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.179361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.179666 (+ 305us) server_negotiation.cc:407] Connection header received
0620 12:18:46.179839 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.179843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.179899 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.179975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.180458 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45998: BlockingRecv error: recv got EOF from 127.0.0.1:45998 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 12:18:46.199935 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.198886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.24.141.61:35575)
0620 12:18:46.199077 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.199082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.199099 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:46.199248 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.199251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.199298 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.199365 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.199815 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46008: BlockingRecv error: recv got EOF from 127.0.0.1:46008 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 12:18:46.236126 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.234766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.24.141.61:35575)
0620 12:18:46.234956 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.234961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.235049 (+ 88us) server_negotiation.cc:407] Connection header received
0620 12:18:46.235208 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.235213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.235297 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.235434 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.235968 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46018: BlockingRecv error: recv got EOF from 127.0.0.1:46018 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 12:18:46.303977 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.302889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.24.141.61:35575)
0620 12:18:46.303073 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.303079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.303097 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:18:46.303214 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.303218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.303273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.303352 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.303853 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46024: BlockingRecv error: recv got EOF from 127.0.0.1:46024 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 12:18:46.436589 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.435110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.24.141.61:35575)
0620 12:18:46.435321 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.435327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.435458 (+ 131us) server_negotiation.cc:407] Connection header received
0620 12:18:46.435711 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.435714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.435773 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.435850 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.436444 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46026: BlockingRecv error: recv got EOF from 127.0.0.1:46026 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 12:18:46.696509 15113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:46.695219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.24.141.61:35575)
0620 12:18:46.695405 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:46.695410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:46.695473 (+ 63us) server_negotiation.cc:407] Connection header received
0620 12:18:46.695690 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:46.695693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:46.695752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:46.695832 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:46.696369 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46036: BlockingRecv error: recv got EOF from 127.0.0.1:46036 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:18:46.899334 15200 catalog_manager.cc:1259] Loaded cluster ID: d9e5bd50a90e4f97819f6ea06d92c46d
I20260620 12:18:46.899439 15200 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: loading cluster ID for follower catalog manager: success
I20260620 12:18:46.902292 15200 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: acquiring CA information for follower catalog manager: success
I20260620 12:18:46.904276 15200 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f3af093d9214f3a9d94e5bd15b2bf41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:18:47.049999 15244 catalog_manager.cc:1259] Loaded cluster ID: d9e5bd50a90e4f97819f6ea06d92c46d
I20260620 12:18:47.050096 15244 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: loading cluster ID for follower catalog manager: success
I20260620 12:18:47.052117 15244 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: acquiring CA information for follower catalog manager: success
I20260620 12:18:47.054030 15244 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e2b8b2027ff4f0ea3c720951e200992: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:18:47.215883 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:47.211387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.24.141.61:35575)
0620 12:18:47.211761 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:47.211766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:47.211787 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:18:47.211924 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:47.211928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:47.212103 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:47.212238 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:47.212522 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.213187 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.214154 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.214395 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.214496 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:47.214831 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:47.215428 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:47.215551 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:47.215610 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 12:18:47.215687 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":85,"threads_started":1}
I20260620 12:18:47.224416 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:47.219833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.24.141.60:40745)
0620 12:18:47.220105 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:47.220109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:47.220126 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:47.220331 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:47.220334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:47.220483 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:47.220604 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:47.220912 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.221502 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.222482 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.222757 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.222845 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:47.223263 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:47.223882 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:47.223986 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:47.224165 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:18:47.224228 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260620 12:18:47.232559 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:47.228311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.24.141.60:40745)
0620 12:18:47.228440 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:47.228444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:47.228627 (+ 183us) server_negotiation.cc:407] Connection header received
0620 12:18:47.228797 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:47.228801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:47.228857 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:47.228900 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:47.229302 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.230102 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.231211 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.231519 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.231572 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:47.231928 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:47.232008 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:47.232072 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:47.232267 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 12:18:47.232393 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:18:47.232746 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:47.227928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.24.141.61:35575)
0620 12:18:47.228083 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:47.228088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:47.228345 (+ 257us) server_negotiation.cc:407] Connection header received
0620 12:18:47.228587 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:47.228591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:47.228661 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:47.228739 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:47.229006 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.229707 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.230831 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.231193 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.231322 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:47.232085 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:47.232175 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:47.232238 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:47.232512 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 12:18:47.232581 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:18:47.233451 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:47.227920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.24.141.62:41025)
0620 12:18:47.228282 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:47.228300 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:47.228634 (+ 334us) server_negotiation.cc:407] Connection header received
0620 12:18:47.228820 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:47.228823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:47.229024 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:47.229201 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:47.229449 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.230243 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.231365 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:47.231647 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:47.231749 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:47.232121 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:47.232642 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:47.232727 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:47.233201 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 12:18:47.233281 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":91,"threads_started":1}
I20260620 12:18:47.237164 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 14965
I20260620 12:18:47.244318 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 15040
I20260620 12:18:47.251773 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 15114
2026-06-20T12:18:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3309 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:18:47 dist-test-slave-rh5v krb5kdc[15330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:18:47 dist-test-slave-rh5v krb5kdc[15330](info): set up 2 sockets
Jun 20 12:18:47 dist-test-slave-rh5v krb5kdc[15330](info): commencing operation
krb5kdc: starting...
W20260620 12:18:48.230031 25140 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/test-user.keytab.
Jun 20 12:18:48 dist-test-slave-rh5v krb5kdc[15330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:18:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:18:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.141.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
Entry for principal kudu/127.24.141.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab.
I20260620 12:18:48.364984 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44657
--webserver_interface=127.24.141.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:44657,127.24.141.61:46159,127.24.141.60:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:18:48.501045 15343 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:18:48.501515 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:48.501542 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:48.501590 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:48.506170 15343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:48.506274 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:48.506294 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:48.506310 15343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:48.506325 15343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:48.512684 15343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:44657,127.24.141.61:46159,127.24.141.60:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.62.keytab
--principal=kudu/127.24.141.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.62:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.141.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:48.514271 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:48.515571 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:48.526304 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:48.526310 15348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:48.526304 15349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:48.526544 15343 server_base.cc:1034] running on GCE node
I20260620 12:18:48.527657 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:48.529314 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:48.530565 15343 hybrid_clock.cc:648] HybridClock initialized: now 1781957928530537 us; error 60 us; skew 500 ppm
Jun 20 12:18:48 dist-test-slave-rh5v krb5kdc[15330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:48.534292 15343 init.cc:377] Logged in from keytab as kudu/127.24.141.62@KRBTEST.COM (short username kudu)
I20260620 12:18:48.535676 15343 webserver.cc:460] Webserver started at http://127.24.141.62:40495/ using document root <none> and password file <none>
I20260620 12:18:48.536382 15343 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:48.536453 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:48.536716 15343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:48.539382 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data/instance:
uuid: "38074d8365ca44278e3ab5c453b73662"
format_stamp: "Formatted at 2026-06-20 12:18:48 on dist-test-slave-rh5v"
server_key: "3bb4ea64e0955bca86714b691ae4bd8d"
server_key_iv: "4921d364520a1f4b00d01b55681e73f3"
server_key_version: "encryptionkey@0"
I20260620 12:18:48.540086 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/wal/instance:
uuid: "38074d8365ca44278e3ab5c453b73662"
format_stamp: "Formatted at 2026-06-20 12:18:48 on dist-test-slave-rh5v"
server_key: "3bb4ea64e0955bca86714b691ae4bd8d"
server_key_iv: "4921d364520a1f4b00d01b55681e73f3"
server_key_version: "encryptionkey@0"
I20260620 12:18:48.545101 15343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 12:18:48.548520 15358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:48.550506 15343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 12:18:48.550699 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/wal
uuid: "38074d8365ca44278e3ab5c453b73662"
format_stamp: "Formatted at 2026-06-20 12:18:48 on dist-test-slave-rh5v"
server_key: "3bb4ea64e0955bca86714b691ae4bd8d"
server_key_iv: "4921d364520a1f4b00d01b55681e73f3"
server_key_version: "encryptionkey@0"
I20260620 12:18:48.550844 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:48.573140 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:48.577478 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:48.577733 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:48.586961 15343 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.62:44657
I20260620 12:18:48.587385 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.62:44657 every 8 connection(s)
I20260620 12:18:48.588385 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/data/info.pb
I20260620 12:18:48.592299 15411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:48.594645 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 15343
I20260620 12:18:48.594792 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-0/wal/instance
I20260620 12:18:48.595151 25140 external_mini_cluster.cc:1428] Setting key 119ec04ecabf71e0ac5b614330ce97a7
I20260620 12:18:48.598043 15411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.141.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.61@KRBTEST.COM" created.
Jun 20 12:18:48 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:48.620730 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.601366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.24.141.62:44657)
0620 12:18:48.602150 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:48.602172 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:48.602561 (+ 389us) server_negotiation.cc:407] Connection header received
0620 12:18:48.604554 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:48.604561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:48.604921 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:48.605195 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:48.606346 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.607488 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.609315 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.609841 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.610149 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.613854 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:48.613867 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:48.613893 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:48.613943 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:48.616431 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.617351 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.617359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.617362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.617552 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.617967 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.617970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.617972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.618558 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:48.618690 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:48.619222 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 12:18:48.619528 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":608,"spinlock_wait_cycles":57088,"thread_start_us":199,"threads_started":1}
I20260620 12:18:48.620735 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.601366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44657 (local address 127.0.0.1:39830)
0620 12:18:48.602246 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.602340 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:48.603974 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:48.605240 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:48.605247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:48.605711 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:48.606114 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.606132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.607740 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.607745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.609088 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.609107 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.610155 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.610159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.610314 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.611133 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:48.611159 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:48.613545 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:48.616680 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.616698 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.616710 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.617165 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.617667 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.617669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.617671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.617834 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.618702 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:48.618710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:48.618916 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 12:18:48.619127 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 12:18:48.619671 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":112,"thread_start_us":192,"threads_started":1}
I20260620 12:18:48.624390 15411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:48.625469 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.625023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60748)
0620 12:18:48.625235 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.625356 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab.
Entry for principal kudu/127.24.141.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab.
W20260620 12:18:48.625746 15362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
I20260620 12:18:48.626893 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:46159
--webserver_interface=127.24.141.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab
--principal=kudu/127.24.141.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:44657,127.24.141.61:46159,127.24.141.60:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:18:48.626935 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.626687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60762)
0620 12:18:48.626809 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.626866 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 12:18:48.627107 15411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:46159: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
I20260620 12:18:48.676425 15411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } attempt: 1
I20260620 12:18:48.677560 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.677221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60776)
0620 12:18:48.677391 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.677460 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 12:18:48.678412 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.678169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60782)
0620 12:18:48.678287 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.678344 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:18:48.678699 15411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:46159: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
I20260620 12:18:48.746932 15411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } attempt: 2
I20260620 12:18:48.748059 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.747700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60798)
0620 12:18:48.747872 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.747944 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:18:48.749235 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.748876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60810)
0620 12:18:48.749053 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.749159 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 12:18:48.749396 15411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:46159: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
W20260620 12:18:48.761713 15418 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:18:48.762060 15418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:48.762085 15418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:48.762130 15418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:48.767040 15418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:48.767145 15418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:48.767167 15418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:48.767185 15418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:48.767200 15418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:48.774394 15418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:44657,127.24.141.61:46159,127.24.141.60:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.61.keytab
--principal=kudu/127.24.141.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.61:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.141.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:48.775764 15418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:48.776957 15418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:48.787253 15426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:48.787256 15424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:48.787297 15423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:48.788086 15418 server_base.cc:1034] running on GCE node
I20260620 12:18:48.788858 15418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:48.790504 15418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:48.791687 15418 hybrid_clock.cc:648] HybridClock initialized: now 1781957928791675 us; error 59 us; skew 500 ppm
Jun 20 12:18:48 dist-test-slave-rh5v krb5kdc[15330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:48.795368 15418 init.cc:377] Logged in from keytab as kudu/127.24.141.61@KRBTEST.COM (short username kudu)
I20260620 12:18:48.796613 15418 webserver.cc:460] Webserver started at http://127.24.141.61:37729/ using document root <none> and password file <none>
I20260620 12:18:48.797390 15418 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:48.797477 15418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:48.797727 15418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:48.800475 15418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data/instance:
uuid: "9c8818919eb4438fbb1c31a93f1ff26f"
format_stamp: "Formatted at 2026-06-20 12:18:48 on dist-test-slave-rh5v"
server_key: "698a1b084207c9152fdb42302dbca2e0"
server_key_iv: "cfd6095de57d0739ba9cb5fd2e5667fc"
server_key_version: "encryptionkey@0"
I20260620 12:18:48.801285 15418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/wal/instance:
uuid: "9c8818919eb4438fbb1c31a93f1ff26f"
format_stamp: "Formatted at 2026-06-20 12:18:48 on dist-test-slave-rh5v"
server_key: "698a1b084207c9152fdb42302dbca2e0"
server_key_iv: "cfd6095de57d0739ba9cb5fd2e5667fc"
server_key_version: "encryptionkey@0"
I20260620 12:18:48.806574 15418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:18:48.810045 15433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:48.811787 15418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:18:48.811982 15418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/wal
uuid: "9c8818919eb4438fbb1c31a93f1ff26f"
format_stamp: "Formatted at 2026-06-20 12:18:48 on dist-test-slave-rh5v"
server_key: "698a1b084207c9152fdb42302dbca2e0"
server_key_iv: "cfd6095de57d0739ba9cb5fd2e5667fc"
server_key_version: "encryptionkey@0"
I20260620 12:18:48.812151 15418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:48.833039 15418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:48.837538 15418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:48.837805 15418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:48.840553 15411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } attempt: 3
I20260620 12:18:48.842557 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.841266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60822)
0620 12:18:48.842287 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.842409 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":868}
I20260620 12:18:48.843683 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.843402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60836)
0620 12:18:48.843544 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.843607 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 12:18:48.843930 15411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.61:46159: Network error: Client connection negotiation failed: client connection to 127.24.141.61:46159: connect: Connection refused (error 111)
I20260620 12:18:48.847834 15418 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.61:46159
I20260620 12:18:48.848208 15485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.61:46159 every 8 connection(s)
I20260620 12:18:48.849082 15418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/data/info.pb
I20260620 12:18:48.853184 15486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:48.856755 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 15418
I20260620 12:18:48.856901 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-1/wal/instance
I20260620 12:18:48.857283 25140 external_mini_cluster.cc:1428] Setting key 43a03122682de33f05f1681a079688ca
I20260620 12:18:48.858872 15486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.141.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.141.60@KRBTEST.COM" created.
Jun 20 12:18:48 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:48.877714 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.861251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.24.141.62:44657)
0620 12:18:48.861471 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:48.861477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:48.862864 (+ 1387us) server_negotiation.cc:407] Connection header received
0620 12:18:48.864359 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:48.864364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:48.864459 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:48.864607 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:48.866589 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.867264 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.869507 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.869784 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.869897 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.873696 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:48.873700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:48.873703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:48.873737 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:48.875323 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.875986 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.875989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.875990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.876068 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.876430 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.876433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.876435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.876649 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:48.876733 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:48.877379 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 12:18:48.877535 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 12:18:48.878628 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.861647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44657 (local address 127.0.0.1:39842)
0620 12:18:48.862583 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.862674 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:48.863917 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:48.864899 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:48.864910 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:48.865481 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:48.866352 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.866387 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.867510 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.867515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.869307 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.869332 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.869987 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.869992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.870223 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.871164 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:48.871197 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:48.873388 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:48.875497 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.875505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.875523 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.875861 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.876158 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.876161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.876163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.876320 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.876755 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:48.876763 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:48.877106 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 12:18:48.877293 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 12:18:48.877695 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":218,"threads_started":1}
I20260620 12:18:48.880694 15486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.141.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab.
Entry for principal kudu/127.24.141.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab.
Jun 20 12:18:48 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:18:48.887604 25140 external_mini_cluster.cc:1326] Running /tmp/dist-test-task23Ce8q/build/debug/bin/kudu
/tmp/dist-test-task23Ce8q/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:42699
--webserver_interface=127.24.141.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab
--principal=kudu/127.24.141.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.141.20:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.141.62:44657,127.24.141.61:46159,127.24.141.60:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:18:48.891454 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.881446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60848)
0620 12:18:48.881612 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.881630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:48.881746 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:48.882421 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:48.882424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:48.882653 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:48.882768 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.882774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.883813 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.883816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.884372 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.884380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.885020 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.885023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.885266 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.885788 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:48.885813 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:48.887454 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:48.889805 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.889809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.889811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.890023 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.890320 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.890323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.890325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.890398 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.890891 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:48.890900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:48.890988 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 12:18:48.891075 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:18:48.891248 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":128}
I20260620 12:18:48.891472 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.881578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.24.141.61:46159)
0620 12:18:48.881971 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:48.881984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:48.882010 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:18:48.882057 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:48.882065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:48.882293 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:48.882438 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:48.882873 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.883700 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.884478 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.884908 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.885003 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.887650 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:48.887666 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:48.887683 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:48.887711 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:48.889652 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.890137 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.890143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.890144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.890232 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.890478 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.890481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.890483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.890786 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:48.890925 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:48.891177 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 12:18:48.891325 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":290,"thread_start_us":146,"threads_started":1}
I20260620 12:18:48.894526 15486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:48.895558 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.895191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55914)
0620 12:18:48.895362 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.895466 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 12:18:48.895826 15436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
I20260620 12:18:48.897169 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.896821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55920)
0620 12:18:48.896982 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.897044 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 12:18:48.897356 15486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:42699: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
I20260620 12:18:48.946668 15486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } attempt: 1
I20260620 12:18:48.947855 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.947429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55930)
0620 12:18:48.947659 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.947746 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 12:18:48.948786 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.948526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55934)
0620 12:18:48.948671 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.948722 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 12:18:48.948925 15486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:42699: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
I20260620 12:18:48.987180 15411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } attempt: 4
Jun 20 12:18:48 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:18:48.998129 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.988244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.24.141.61:46159)
0620 12:18:48.988397 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:48.988402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:48.988511 (+ 109us) server_negotiation.cc:407] Connection header received
0620 12:18:48.988671 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:48.988675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:48.988725 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:48.988797 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:48.989355 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.990393 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.991651 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.991944 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.992089 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.994833 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:48.994836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:48.994839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:48.994868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:48.996298 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.996787 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.996790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.996792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.996884 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:48.997227 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:48.997231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:48.997233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:48.997454 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:48.997554 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:48.997837 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 12:18:48.997960 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 12:18:48.998129 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:48.988111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60864)
0620 12:18:48.988399 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:48.988419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:48.988567 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:48.988857 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:48.988860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:48.989099 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:48.989220 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.989227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.990529 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.990533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.991311 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:48.991335 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:48.992141 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:48.992146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:48.992269 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:48.992812 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:48.992825 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:48.994621 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:48.996449 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.996453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.996456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.996686 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.996972 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:48.996975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:48.996978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:48.997100 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:48.997558 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:48.997562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:48.997666 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 12:18:48.997750 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 12:18:48.997939 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260620 12:18:48.999571 15411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:49.000567 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.000154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55942)
0620 12:18:49.000343 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.000442 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 12:18:49.001576 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.001305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55952)
0620 12:18:49.001447 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.001510 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 12:18:49.001817 15411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:42699: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
I20260620 12:18:49.017099 15486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } attempt: 2
I20260620 12:18:49.018218 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.017771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55964)
0620 12:18:49.018029 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.018120 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 12:18:49.019414 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.018982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55974)
0620 12:18:49.019145 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.019320 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 12:18:49.019609 15486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:42699: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
W20260620 12:18:49.022488 15492 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:18:49.022917 15492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:18:49.022959 15492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:18:49.023028 15492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:18:49.027663 15492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:18:49.027751 15492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:18:49.027779 15492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:18:49.027803 15492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:18:49.027834 15492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:18:49.034077 15492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.141.20:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.141.62:44657,127.24.141.61:46159,127.24.141.60:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/krb5kdc/kudu_127.24.141.60.keytab
--principal=kudu/127.24.141.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.141.60:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.141.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:18:49.035621 15492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:18:49.036912 15492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:18:49.047093 15498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:49.047097 15499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:18:49.047097 15501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:18:49.047449 15492 server_base.cc:1034] running on GCE node
I20260620 12:18:49.048259 15492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:18:49.049664 15492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:18:49.050879 15492 hybrid_clock.cc:648] HybridClock initialized: now 1781957929050852 us; error 50 us; skew 500 ppm
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957929, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:18:49.054448 15492 init.cc:377] Logged in from keytab as kudu/127.24.141.60@KRBTEST.COM (short username kudu)
I20260620 12:18:49.055660 15492 webserver.cc:460] Webserver started at http://127.24.141.60:35233/ using document root <none> and password file <none>
I20260620 12:18:49.056413 15492 fs_manager.cc:359] Metadata directory not provided
I20260620 12:18:49.056501 15492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:18:49.056721 15492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:18:49.059206 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data/instance:
uuid: "9888042db00342af8d6edf308070cbec"
format_stamp: "Formatted at 2026-06-20 12:18:49 on dist-test-slave-rh5v"
server_key: "d73b3de99ac2d358b22b0db52af9e34f"
server_key_iv: "fe31596837ea23f8d2df481c12c02c42"
server_key_version: "encryptionkey@0"
I20260620 12:18:49.060045 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/wal/instance:
uuid: "9888042db00342af8d6edf308070cbec"
format_stamp: "Formatted at 2026-06-20 12:18:49 on dist-test-slave-rh5v"
server_key: "d73b3de99ac2d358b22b0db52af9e34f"
server_key_iv: "fe31596837ea23f8d2df481c12c02c42"
server_key_version: "encryptionkey@0"
I20260620 12:18:49.061053 15411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } attempt: 1
I20260620 12:18:49.062057 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.061707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55990)
0620 12:18:49.061882 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.061948 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 12:18:49.063112 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.062877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:55998)
0620 12:18:49.062995 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.063050 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 12:18:49.063285 15411 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.141.60:42699: Network error: Client connection negotiation failed: client connection to 127.24.141.60:42699: connect: Connection refused (error 111)
I20260620 12:18:49.065728 15492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 12:18:49.069000 15508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:18:49.070827 15492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:18:49.071039 15492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data,/tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/wal
uuid: "9888042db00342af8d6edf308070cbec"
format_stamp: "Formatted at 2026-06-20 12:18:49 on dist-test-slave-rh5v"
server_key: "d73b3de99ac2d358b22b0db52af9e34f"
server_key_iv: "fe31596837ea23f8d2df481c12c02c42"
server_key_version: "encryptionkey@0"
I20260620 12:18:49.071170 15492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:18:49.088151 15492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:18:49.092329 15492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:18:49.092531 15492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:18:49.101255 15492 rpc_server.cc:307] RPC server started. Bound to: 127.24.141.60:42699
I20260620 12:18:49.101585 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.141.60:42699 every 8 connection(s)
I20260620 12:18:49.102662 15492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/data/info.pb
I20260620 12:18:49.106240 15561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:18:49.106457 25140 external_mini_cluster.cc:1388] Started /tmp/dist-test-task23Ce8q/build/debug/bin/kudu as pid 15492
I20260620 12:18:49.106599 25140 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task23Ce8q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957618728254-25140-0/minicluster-data/master-2/wal/instance
I20260620 12:18:49.106983 25140 external_mini_cluster.cc:1428] Setting key fd1117c3b0e8f9729801279f00d3c965
I20260620 12:18:49.110800 15486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } attempt: 3
I20260620 12:18:49.112239 15561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.61@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957929, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:49.124804 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.111395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:56020)
0620 12:18:49.111720 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.111738 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:49.111842 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:49.112623 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:49.112626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:49.112824 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:49.112925 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.112931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.114295 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.114300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.115051 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.115068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.116243 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.116246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.116344 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.117172 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:49.117187 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:49.119397 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:49.122210 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.122215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.122218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.122478 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.122989 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.122993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.122995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.123103 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.123884 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:49.123888 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:49.124020 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 12:18:49.124122 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 12:18:49.124321 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260620 12:18:49.126092 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.111937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.24.141.60:42699)
0620 12:18:49.112376 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.112386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.112399 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:18:49.112442 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.112444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.112490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.112575 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.113039 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.114098 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.115244 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.116073 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.116609 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.119585 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.119593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.119610 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.119644 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.122059 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.122602 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.122609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.122618 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.122811 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.123219 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.123223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.123225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.123747 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.123909 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.124321 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.124703 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":364,"server-negotiator.queue_time_us":356,"thread_start_us":176,"threads_started":1}
I20260620 12:18:49.126132 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.108731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.24.141.60:42699)
0620 12:18:49.109341 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.109364 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.109417 (+ 53us) server_negotiation.cc:407] Connection header received
0620 12:18:49.110281 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.110311 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.110862 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.111374 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.112365 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.114105 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.115438 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.116066 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.116233 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.119646 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.119650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.119653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.119692 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.122102 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.122639 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.122643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.122645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.122816 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.123120 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.123123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.123125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.123751 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.123967 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.124855 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.124996 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":407,"thread_start_us":157,"threads_started":1}
I20260620 12:18:49.127461 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.114649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.24.141.62:44657)
0620 12:18:49.114803 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.114807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.115290 (+ 483us) server_negotiation.cc:407] Connection header received
0620 12:18:49.115551 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.115555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.115624 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.115717 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.116752 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.117470 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.118735 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.118991 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.119114 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.122705 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.122710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.122713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.122748 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.125381 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.125936 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.125940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.125942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.126049 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.126356 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.126360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.126362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.126601 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.126676 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.127143 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.127292 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:18:49.127986 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.114565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.62:44657 (local address 127.0.0.1:39848)
0620 12:18:49.115145 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.115221 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:49.115397 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:49.115856 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:49.115863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:49.116438 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:49.116613 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.116621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.117587 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.117591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.118536 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.118549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.119304 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.119307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.119420 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.119981 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:49.119993 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:49.122461 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:49.125519 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.125527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.125534 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.125817 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.126168 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.126170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.126172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.126257 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.126752 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:49.126758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:49.126886 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 12:18:49.127628 (+ 742us) client_negotiation.cc:240] Negotiation successful
0620 12:18:49.127832 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":102,"threads_started":1}
I20260620 12:18:49.129491 15561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:18:49.130452 15411 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } attempt: 2
I20260620 12:18:49.132071 15486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: Bootstrap starting.
I20260620 12:18:49.135142 15486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: Neither blocks nor log segments found. Creating new log.
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.62@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
I20260620 12:18:49.136251 15486 log.cc:824] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:49.139189 15486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: No bootstrap required, opened a new log
I20260620 12:18:49.140007 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.130907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:56022)
0620 12:18:49.131052 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.131068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:49.131167 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:49.131398 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:49.131400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:49.131593 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:49.131698 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.131703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.132496 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.132499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.133090 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.133101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.133568 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.133571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.133649 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.134133 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:49.134145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:49.135894 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:49.138032 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.138036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.138038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.138347 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.138659 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.138662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.138663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.138755 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.139178 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:49.139182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:49.139312 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 12:18:49.139398 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:18:49.139565 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957929, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:18:49.141726 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.130975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.24.141.60:42699)
0620 12:18:49.131077 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.131080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.131115 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:18:49.131253 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.131256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.131300 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.131365 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.131807 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.132394 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.133210 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.133460 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.133549 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.136074 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.136078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.136080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.136109 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.137902 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.138473 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.138476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.138478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.138565 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.138857 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.138860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.138861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.139076 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.141190 (+ 2114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.141356 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.141552 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.62@KRBTEST.COM
I20260620 12:18:49.144040 15486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.144335 15486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8818919eb4438fbb1c31a93f1ff26f, State: Initialized, Role: FOLLOWER
I20260620 12:18:49.144761 15486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.145557 15568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.145905 15568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:49.145961 15411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662: Bootstrap starting.
I20260620 12:18:49.146587 15486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:49.146736 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.136116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.24.141.62:44657)
0620 12:18:49.136249 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.136253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.136267 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:18:49.136306 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.136309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.136354 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.136430 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.137476 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.138223 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.139333 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.139639 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.139713 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.143073 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.143079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.143083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.143118 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.144546 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.145051 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.145055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.145088 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.145172 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.145502 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.145506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.145508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.145756 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.145983 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.146370 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.146498 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:18:49.149448 15411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:49.150408 15411 log.cc:824] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:49.152999 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.130202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.24.141.61:46159)
0620 12:18:49.135673 (+ 5471us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.135680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.135699 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:18:49.135752 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.135756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.135814 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.135917 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.136630 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.137557 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.138652 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.138979 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.139060 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.141764 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.141768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.141771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.141807 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.143721 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.151471 (+ 7750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.151476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.151478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.151596 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.151899 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.151903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.151905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.152183 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.152252 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.152641 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.152795 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5361}
I20260620 12:18:49.153606 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.130110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.61:46159 (local address 127.0.0.1:60874)
0620 12:18:49.130251 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.130270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:49.130359 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:49.135978 (+ 5619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:49.135982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:49.136197 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:49.136347 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.136460 (+ 113us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.137735 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.137739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.138516 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.138524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.139227 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.139230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.139439 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.139882 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:49.139896 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:49.141573 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:49.150995 (+ 9422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.150998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.151002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.151337 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.151710 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.151713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.151715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.151801 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.152324 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:49.152326 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:49.152434 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 12:18:49.153249 (+ 815us) client_negotiation.cc:240] Negotiation successful
0620 12:18:49.153443 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:18:49.153887 15580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:49.154008 15580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:49.154398 15561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.141.61@KRBTEST.COM
I20260620 12:18:49.151026 15418 master_runner.cc:186] Error getting master registration for 127.24.141.62:44657: OK, instance_id { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" instance_seqno: 1781957928584323 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:18:49.157423 15418 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:18:49.157660 15411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662: No bootstrap required, opened a new log
I20260620 12:18:49.160017 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.148193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.24.141.61:46159)
0620 12:18:49.148549 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.148558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.148573 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:18:49.148618 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.148621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.148685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.148787 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.149697 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.150414 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.151685 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.152061 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.152159 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.155587 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.155591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.155594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.155621 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.157517 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.158045 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.158048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.158050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.158148 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.158436 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.158451 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.158452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.158672 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.158885 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.159673 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.159818 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
Jun 20 12:18:49 dist-test-slave-rh5v krb5kdc[15330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957929, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.141.60@KRBTEST.COM for kudu/127.24.141.60@KRBTEST.COM
I20260620 12:18:49.163700 25140 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:18:49.163219 15411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.163970 15411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38074d8365ca44278e3ab5c453b73662, State: Initialized, Role: FOLLOWER
I20260620 12:18:49.164455 15411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.165130 15385 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "38074d8365ca44278e3ab5c453b73662"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862
I20260620 12:18:49.165948 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.155592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.24.141.60:42699)
0620 12:18:49.155714 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.155719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.156432 (+ 713us) server_negotiation.cc:407] Connection header received
0620 12:18:49.156772 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.156775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.156827 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.156898 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.157450 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.158141 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.159176 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.159497 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.159575 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.162308 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.162313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.162316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.162342 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.164059 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.164647 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.164651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.164653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.164731 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.164986 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.164989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.164991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.165255 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.165328 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.165625 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.165764 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:18:49.166497 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.155056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.141.60:42699 (local address 127.0.0.1:56028)
0620 12:18:49.156356 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 12:18:49.156369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:18:49.156464 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:18:49.157006 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:18:49.157008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:18:49.157250 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:18:49.157341 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.157348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.158274 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.158277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.159054 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:18:49.159064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.159740 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.159743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:18:49.159818 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.160359 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:18:49.160374 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:18:49.162113 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:18:49.164232 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.164235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.164238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.164533 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.164821 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:18:49.164824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:18:49.164826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:18:49.164903 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:18:49.165384 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:18:49.165386 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:18:49.165467 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 12:18:49.166199 (+ 732us) client_negotiation.cc:240] Negotiation successful
0620 12:18:49.166354 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219}
I20260620 12:18:49.166564 15411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:49.166823 15385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 12:18:49.166953 15385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:49.168654 15385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.170140 15583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.170290 15583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:49.170336 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: Bootstrap starting.
I20260620 12:18:49.170789 15460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38074d8365ca44278e3ab5c453b73662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c8818919eb4438fbb1c31a93f1ff26f"
I20260620 12:18:49.171133 15460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:49.171891 15385 leader_election.cc:288] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [CANDIDATE]: Term 1 election: Requested vote from peers 9c8818919eb4438fbb1c31a93f1ff26f (127.24.141.61:46159), 9888042db00342af8d6edf308070cbec (127.24.141.60:42699)
I20260620 12:18:49.172927 15460 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38074d8365ca44278e3ab5c453b73662 in term 1.
I20260620 12:18:49.173571 15362 leader_election.cc:302] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38074d8365ca44278e3ab5c453b73662, 9c8818919eb4438fbb1c31a93f1ff26f; no voters:
I20260620 12:18:49.173723 15561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: Neither blocks nor log segments found. Creating new log.
I20260620 12:18:49.174722 15561 log.cc:824] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: Log is configured to *not* fsync() on all Append() calls
I20260620 12:18:49.175171 15583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:18:49.175442 15583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [term 1 LEADER]: Becoming Leader. State: Replica: 38074d8365ca44278e3ab5c453b73662, State: Running, Role: LEADER
I20260620 12:18:49.175901 15583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.179544 15600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38074d8365ca44278e3ab5c453b73662. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.179849 15600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: This master's current role is: LEADER
W20260620 12:18:49.181262 15594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:49.183925 15594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:49.184101 15603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:18:49.185276 15603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:18:49.182744 15343 master_runner.cc:186] Error getting master registration for 127.24.141.60:42699: OK, instance_id { permanent_uuid: "9888042db00342af8d6edf308070cbec" instance_seqno: 1781957929098421 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:18:49.186679 15343 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:18:49.195318 15460 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [term 1 FOLLOWER]: Refusing update from remote peer 38074d8365ca44278e3ab5c453b73662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:49.195477 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: No bootstrap required, opened a new log
I20260620 12:18:49.196314 15600 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:49.199717 15561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.200044 15561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9888042db00342af8d6edf308070cbec, State: Initialized, Role: FOLLOWER
I20260620 12:18:49.200412 15561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } }
I20260620 12:18:49.201004 15568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38074d8365ca44278e3ab5c453b73662. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.201167 15568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:49.202101 15561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:18:49.202169 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38074d8365ca44278e3ab5c453b73662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9888042db00342af8d6edf308070cbec"
I20260620 12:18:49.202446 15535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:18:49.204625 15535 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38074d8365ca44278e3ab5c453b73662 in term 1.
I20260620 12:18:49.205399 15534 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [term 1 FOLLOWER]: Refusing update from remote peer 38074d8365ca44278e3ab5c453b73662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:18:49.205621 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.182760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.24.141.60:42699)
0620 12:18:49.182895 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.182900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.187029 (+ 4129us) server_negotiation.cc:407] Connection header received
0620 12:18:49.187192 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.187196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.187267 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.187386 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.196709 (+ 9323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.197501 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.198664 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.198969 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.199047 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.201553 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.201558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.201561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.201593 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.203434 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.203963 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.203967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.203969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.204053 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.204326 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.204330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.204331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.204583 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.204740 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.205239 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.205379 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:18:49.206619 15600 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:18:49.212821 15600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.212975 15600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:49.214651 15568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.214799 15568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:49.215368 15607 mvcc.cc:205] Tried to move back new op lower bound from 7298899677969678336 to 7298899677908701184. Current Snapshot: MvccSnapshot[applied={T|T < 7298899677969678336}]
I20260620 12:18:49.217433 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.217609 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:49.217717 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38074d8365ca44278e3ab5c453b73662. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.217813 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:49.217901 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.183412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.24.141.61:46159)
0620 12:18:49.183549 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.183553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.188981 (+ 5428us) server_negotiation.cc:407] Connection header received
0620 12:18:49.191513 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.191517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.191590 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.191647 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.194541 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.195285 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.207662 (+ 12377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.208055 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.208142 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.211067 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.211072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.211075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.211111 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.212707 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.213912 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.213916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.213918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.214022 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.216138 (+ 2116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.216141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.216143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.216411 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.216547 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.217574 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.217729 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:18:49.218763 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:49.181800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.24.141.62:44657)
0620 12:18:49.181935 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.181951 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.188758 (+ 6807us) server_negotiation.cc:407] Connection header received
0620 12:18:49.190284 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.190288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.190348 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.190405 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.192612 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.193377 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.205733 (+ 12356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:49.206129 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:49.206219 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:49.213535 (+ 7316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:18:49.213539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:18:49.213542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:18:49.213576 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:18:49.215265 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.215748 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.215751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.215753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.215839 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:18:49.216871 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:18:49.216874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:18:49.216876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:18:49.217135 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:18:49.217244 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:49.218496 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0620 12:18:49.218623 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:18:49.218997 15603 catalog_manager.cc:1347] Generated new cluster ID: 6dbd23e174bb44db8a002a6590195a1b
I20260620 12:18:49.219049 15603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260620 12:18:49.221896 15621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:18:49.222040 15621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:18:49.230096 15583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.230320 15583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662 [sys.catalog]: This master's current role is: LEADER
I20260620 12:18:49.231518 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38074d8365ca44278e3ab5c453b73662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38074d8365ca44278e3ab5c453b73662" member_type: VOTER last_known_addr { host: "127.24.141.62" port: 44657 } } peers { permanent_uuid: "9c8818919eb4438fbb1c31a93f1ff26f" member_type: VOTER last_known_addr { host: "127.24.141.61" port: 46159 } } peers { permanent_uuid: "9888042db00342af8d6edf308070cbec" member_type: VOTER last_known_addr { host: "127.24.141.60" port: 42699 } } }
I20260620 12:18:49.231714 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:18:49.232267 15623 mvcc.cc:205] Tried to move back new op lower bound from 7298899677969678336 to 7298899677908701184. Current Snapshot: MvccSnapshot[applied={T|T < 7298899677969678336}]
I20260620 12:18:49.255856 15603 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:18:49.257690 15603 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:18:49.270709 15603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38074d8365ca44278e3ab5c453b73662: Generated new TSK 0
I20260620 12:18:49.272284 15603 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:18:49.314803 15415 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.311344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.24.141.62:44657)
0620 12:18:49.311496 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.311501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.313582 (+ 2081us) server_negotiation.cc:407] Connection header received
0620 12:18:49.313794 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.313798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.313867 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.313921 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.314645 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39876: BlockingRecv error: recv got EOF from 127.0.0.1:39876 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 12:18:49.316512 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.312238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.24.141.61:46159)
0620 12:18:49.312436 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.312442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.315408 (+ 2966us) server_negotiation.cc:407] Connection header received
0620 12:18:49.315597 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.315600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.315660 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.315712 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.316336 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60898: BlockingRecv error: recv got EOF from 127.0.0.1:60898 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 12:18:49.317572 15565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.312888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.24.141.60:42699)
0620 12:18:49.313018 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.313022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.316553 (+ 3531us) server_negotiation.cc:407] Connection header received
0620 12:18:49.316738 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.316742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.316797 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.316846 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.317459 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56050: BlockingRecv error: recv got EOF from 127.0.0.1:56050 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 12:18:49.322603 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.320652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.24.141.61:46159)
0620 12:18:49.320780 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.320785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.321497 (+ 712us) server_negotiation.cc:407] Connection header received
0620 12:18:49.321746 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.321750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.321851 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.321907 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.322468 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60900: BlockingRecv error: recv got EOF from 127.0.0.1:60900 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 12:18:49.328351 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.326565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.24.141.61:46159)
0620 12:18:49.326692 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.326696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.327326 (+ 630us) server_negotiation.cc:407] Connection header received
0620 12:18:49.327536 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.327540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.327594 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.327646 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.328222 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60906: BlockingRecv error: recv got EOF from 127.0.0.1:60906 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 12:18:49.334617 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.332768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.24.141.61:46159)
0620 12:18:49.332907 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.332911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.333602 (+ 691us) server_negotiation.cc:407] Connection header received
0620 12:18:49.333801 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.333805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.333868 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.333919 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.334485 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60918: BlockingRecv error: recv got EOF from 127.0.0.1:60918 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 12:18:49.344267 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.342343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.24.141.61:46159)
0620 12:18:49.342500 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.342504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.343208 (+ 704us) server_negotiation.cc:407] Connection header received
0620 12:18:49.343429 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.343433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.343491 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.343543 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.344144 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60932: BlockingRecv error: recv got EOF from 127.0.0.1:60932 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 12:18:49.358043 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.355508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.24.141.61:46159)
0620 12:18:49.355664 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.355668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.356793 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 12:18:49.357049 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.357053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.357179 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.357239 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.357901 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60940: BlockingRecv error: recv got EOF from 127.0.0.1:60940 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:18:49.379693 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.377676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.24.141.61:46159)
0620 12:18:49.377821 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.377826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.378507 (+ 681us) server_negotiation.cc:407] Connection header received
0620 12:18:49.378755 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.378759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.378825 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.378888 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.379556 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60952: BlockingRecv error: recv got EOF from 127.0.0.1:60952 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 12:18:49.416026 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.414911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.24.141.61:46159)
0620 12:18:49.415104 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.415109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.415126 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:49.415259 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.415262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.415314 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.415382 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.415883 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60966: BlockingRecv error: recv got EOF from 127.0.0.1:60966 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 12:18:49.484537 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.483027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.24.141.61:46159)
0620 12:18:49.483269 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.483274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.483360 (+ 86us) server_negotiation.cc:407] Connection header received
0620 12:18:49.483584 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.483587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.483639 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.483724 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.484420 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60974: BlockingRecv error: recv got EOF from 127.0.0.1:60974 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:18:49.617166 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.615419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.24.141.61:46159)
0620 12:18:49.615631 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.615636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.615845 (+ 209us) server_negotiation.cc:407] Connection header received
0620 12:18:49.615966 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.615969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.616035 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.616171 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.616922 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60990: BlockingRecv error: recv got EOF from 127.0.0.1:60990 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 12:18:49.877532 15574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:18:49.876242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.24.141.61:46159)
0620 12:18:49.876409 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:49.876414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:49.876440 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:18:49.876533 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:49.876537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:49.876618 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:49.876780 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:18:49.877405 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32770: BlockingRecv error: recv got EOF from 127.0.0.1:32770 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:18:50.155681 15580 catalog_manager.cc:1259] Loaded cluster ID: 6dbd23e174bb44db8a002a6590195a1b
I20260620 12:18:50.155783 15580 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: loading cluster ID for follower catalog manager: success
I20260620 12:18:50.157768 15580 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: acquiring CA information for follower catalog manager: success
I20260620 12:18:50.159286 15580 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c8818919eb4438fbb1c31a93f1ff26f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:18:50.223531 15621 catalog_manager.cc:1259] Loaded cluster ID: 6dbd23e174bb44db8a002a6590195a1b
I20260620 12:18:50.223628 15621 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: loading cluster ID for follower catalog manager: success
I20260620 12:18:50.225724 15621 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: acquiring CA information for follower catalog manager: success
I20260620 12:18:50.227133 15621 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9888042db00342af8d6edf308070cbec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:18:50.397321 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:50.392372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.24.141.61:46159)
0620 12:18:50.392840 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:50.392844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:50.392866 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:18:50.392931 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:50.392933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:50.393117 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:50.393248 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:50.393581 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.394205 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.395281 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.395578 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.395721 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:50.396084 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:50.396795 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:50.396943 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:50.397012 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 12:18:50.397132 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":188,"threads_started":1}
I20260620 12:18:50.406244 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:50.401110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.24.141.60:42699)
0620 12:18:50.401477 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:50.401482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:50.401499 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:50.401622 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:50.401625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:50.401810 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:50.401963 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:50.402409 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.403087 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.404206 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.404457 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.404564 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:50.404982 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:50.405661 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:50.405783 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:50.405962 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:18:50.406070 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260620 12:18:50.415179 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:50.410558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.24.141.61:46159)
0620 12:18:50.410851 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:50.410856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:50.411161 (+ 305us) server_negotiation.cc:407] Connection header received
0620 12:18:50.411243 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:50.411247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:50.411313 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:50.411398 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:50.411704 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.412719 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.413869 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.414205 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.414315 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:50.414629 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:50.414709 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:50.414771 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:50.414917 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 12:18:50.414990 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260620 12:18:50.415179 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:50.410699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.24.141.60:42699)
0620 12:18:50.410973 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:50.410978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:50.410995 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:18:50.411061 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:50.411065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:50.411131 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:50.411212 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:50.411525 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.412385 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.413719 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.414050 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.414160 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:50.414584 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:50.414691 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:50.414768 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:50.414904 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 12:18:50.414990 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260620 12:18:50.415567 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:18:50.410175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.24.141.62:44657)
0620 12:18:50.410633 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 12:18:50.410639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:18:50.410657 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:18:50.410808 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:18:50.410812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:18:50.411051 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:18:50.411184 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:18:50.411641 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.412231 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.413655 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:18:50.413987 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:18:50.414092 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:18:50.414491 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:18:50.414932 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:18:50.415013 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:18:50.415304 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 12:18:50.415391 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":99,"threads_started":1}
I20260620 12:18:50.419503 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 15343
I20260620 12:18:50.426378 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 15418
I20260620 12:18:50.433929 25140 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task23Ce8q/build/debug/bin/kudu with pid 15492
2026-06-20T12:18:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3182 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6492 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (311674 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST